CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration

arXiv:2608.30295v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challenging due to the substantial memory requirements and inference latency. In this work, we discover that certain attention heads exhibit sequential consistency in their attention patterns, which can be persistently identified using a coefficient-of-variation-based algorithm. Inspired by this observation, we propose CateKV, a hybrid KV cache method that retains only critical token information for consistent heads, thereby reducing KV cache size and computational overhead, while preserving the majority of KV pairs in adaptive heads to ensure high accuracy. We show the unique characteristics of our algorithm and its extension with existing acceleration methods. Comprehensive evaluations on long-context benchmarks show that, while maintaining accuracy comparable to full attention, CateKV reduces memory usage by up to 2.72imes and accelerates decoding by 2.18imes in single-sample inputs, and boosts throughput by 3.96imes in batch scenarios.

Related

• RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
• Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
• DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
• TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference

Source: arXiv cs.AI | 2026-09-01