RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving

arXiv:2610.18143v1 Announce Type: new Abstract: Prefix reuse improves LLM inference by sharing the KV cache of common prompt prefixes. This helps only if the scheduler can execute a matched branch in the current worker round. Schedulers assume that a matched prefix is ready, which holds for full attention but fails for hybrid LLMs. There, a branch (the consumer) may need recurrent state that another branch (the producer) is still computing in the same round. The native scheduler stops its waiting-queue scan at this blocked consumer, and the resulting head-of-line blocking creates execution bubbles: idle batch slots that independent work could fill. We introduce RadixFlow, which decouples prefix matching from state readiness. Its state-ready traversal turns a global scan barrier into a local wait: only the consumer is blocked until the next round, while feasible independent branches keep filling the batch under the native state check. On five hybrid models, RadixFlow lowers mean latency by 3% to 24% and worker rounds by 11% to 44% at almost unchanged reuse. On the larger Qwen3.5 models, ablations show that most of the gain comes from traversal, which adds little on a full-attention control. Heavier rounds limit how much of this saving reaches latency.

Related

• RKSC: Reasoning-Aware KV Cache Sharing and Confident Early Exit for Multi-Step LLM Inference
• Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs
• PTStore (Prefix Tensor Store): Distributed Prefix Caching and Replication for High Throughput Inference Serving
• Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
• KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse

Source: arXiv cs.AI | 2026-10-15