Research
OmniKVQuant: KV Cache Quantization for Omni-LLMs
arXiv:2609.11582v1 Announce Type: new Abstract: As Omni-modal large language models (Omni-LLMs) take in audio, video and text together, their KV cache memory cost grows. KV cache quantization is the d
arXiv:2609.11582v1 Announce Type: new Abstract: As Omni-modal large language models (Omni-LLMs) take in audio, video and text together, their KV cache memory cost grows. KV cache quantization is the de facto approach in text-only LLMs, but its application to Omni-LLMs remains unexplored. In this paper, we analyze how TurboQuant, a representative rotation-based KV cache quantization method, behaves on multimodal caches and identify two critical issues: temporal key drift and heterogeneous value geometry. To address these, we propose OmniKVQuant, a training-free framework that (i) sets the key quantization range over each short window of the input stream; and (ii) rotates values separately per modality. On Qwen2.5-Omni and Qwen3-Omni, OmniKVQuant enables 2-bit KV caches while substantially preserving performance across seven audio-visual benchmarks. We further provide a fused Triton decode kernel that unpacks the 2-bit cache during attention, so no dense FP16 cache is ever built. Code: https://github.com/kaistmm/OmniKVQuant
Related
- MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering
- Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding
- OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs
Source: arXiv cs.CV | 2026-09-11