GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference

arXiv:2609.17573v1 Announce Type: cross Abstract: Diffusion large language models (dLLMs) are emerging as a promising generative paradigm that complements autoregressive decoding. In long-context settings, KV cache bloat and offloading transfer overhead have become primary bottlenecks in inference systems. Meanwhile, the periodic full-sequence recomputation and localized token updates in dLLMs make the KV lifecycle substantially more dynamic, complicating cache management and prefetch scheduling while making heavyweight token-level indexing or clustering schemes harder to amortize effectively during decoding. To address these challenges, we present extsc{GroupKV}, a lightweight hierarchical KV cache management system for long-context dLLM inference. We observe that under block-wise decoding, tokens within the same generation block tend to access highly overlapping and spatially concentrated context regions, making group-level sparse selection effective. Building on this observation, extsc{GroupKV} partitions the context into contiguous groups and performs coarse-to-fine sparse selection. extsc{GroupKV} further exploits cross-layer consistency to enable predictive prefetching, and incorporates a staleness correction mechanism to maintain cache coherence under dynamic KV updates. Additionally, extsc{GroupKV} adopts streaming prefill to reduce peak memory consumption during prefilling. Experiments show that extsc{GroupKV} extends the maximum serviceable context length by up to 48.00imes under constrained GPU memory, improves end-to-end inference performance by up to 3.73imes in offload-based long-context settings, and maintains competitive task accuracy.

Source: arXiv cs.LG | 2026-09-17