Tailoring the Quantization Space for 1-Bit KV Cache Compression

arXiv:2610.03027v1 Announce Type: cross Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwidth. To mitigate this bottleneck, vector quantization (VQ) has emerged as a promising approach for aggressive KV cache compression. However, existing VQ methods degrade substantially in the 1-bit regime. At such extreme compression, each codebook must represent a larger group of channels with a limited set of centroids, making effective use of its capacity increasingly challenging. To address this, we introduce extbf{TaSQ}, which tailors the VQ target space by combining query-guided channel weighting, cross-head normalization, and covariance-aware channel grouping to better reflect the error sensitivity and statistical structure of cached activations. Since these transforms are RoPE-compatible and can be easily merged into projection weights and codebooks, TaSQ preserves the conventional VQ lookup structure and adds negligible serving overhead. Across general, long-chain-of-thought reasoning, and long-context retrieval benchmarks, TaSQ consistently outperforms existing low-bit KV cache VQ baselines while preserving reasoning stability. On a single RTX 6000 Ada GPU, its SGLang implementation supports up to 14imes larger batch sizes and achieves 1.87imes higher peak throughput compared to the BF16 baseline.

Related

• Interface-Aware KV Cache Quantization for Dense On-Chip NVM in Long-Context LLM Decoding
• HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression
• Output-Aware Rotation for INT2 KV-Cache Quantization
• OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

Source: arXiv cs.AI | 2026-10-05