Safety
Request Order Matters: Cache-History Sensitivity in Selective KV-Cache Reuse for Rolling Agents
arXiv:2610.05833v1 Announce Type: cross Abstract: Long-running agents repeatedly call an LLM while retaining most of their document window, evicting old documents, and appending new ones. These rollin
arXiv:2610.05833v1 Announce Type: cross Abstract: Long-running agents repeatedly call an LLM while retaining most of their document window, evicting old documents, and appending new ones. These rolling updates break exact prefix caching and motivate non-prefix KV-cache reuse with selective recomputation. We show that persistent KV-cache reuse with selective recomputation can be history-dependent: in our rolling-agent workload, an unchanged prompt can produce different answers depending on the requests processed before it. At a matched 5% recomputation budget, document-aligned recomputation reduces answer variation across request orders from 69.0% with CacheBlend's token top-k policy to 26.1%. When each prompt is evaluated after a different sequence of preceding requests, document-aligned recomputation improves fidelity to full prefill by 34.5-52.5 percentage points over token top-k, while both policies achieve approximately 5.7imes median TTFT speedup. Our ablation study shows that, in our rolling-agent workload, contiguity is the main factor associated with robust selective recomputation.
Related
- EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?
- Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders
Source: arXiv cs.LG | 2026-10-06