Research
Evaluating the accuracy of KV cache reuse techniques
arXiv:2609.31415v1 Announce Type: new Abstract: Position-independent KV cache reuse aims to reduce latency in retrieval-augmented generation by reusing chunk-level KV caches across prompts. We show th
arXiv:2609.31415v1 Announce Type: new Abstract: Position-independent KV cache reuse aims to reduce latency in retrieval-augmented generation by reusing chunk-level KV caches across prompts. We show that current evaluations of KV cache reuse techniques rely on measurements that fail to faithfully capture the loss of accuracy attributable to reuse, often artificially inflating the reported effectiveness. We also show that existing datasets do not exhibit the reuse dynamics needed to thoroughly evaluate such techniques. To address these issues, we propose an evaluation methodology that measures this accuracy loss without ambiguity and we introduce Boxoffice, a tool that programmatically generates evaluation datasets that exercise challenging KV cache reuse patterns.
Related
- Shared Global KV with Layer-Specific Local History
- CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG
- HijackKV: New Threat in Position-Independent KV Cache Reuse
- QV-PIC: Query-Aware Visual Position-Independent Caching for Efficient RAG Serving
Source: arXiv cs.LG | 2026-09-28