PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving

arXiv:2607.02525v1 Announce Type: cross Abstract: We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime. PEEK maintains an incremental radix tree over the pending queue, exposing prefix-sharing clusters no existing engine surfaces. A low-overhead dual-walk matches the tree against the engine's prefix cache to yield longest-prefix-match for every waiting request; PEEK then admits cluster pioneers first so siblings inherit the freshly cached prefix, a co-designed eviction hook protects blocks ancestral to queued demand, and a multi-lane stride scheduler bounds starvation. On SGLang and vLLM across five workloads up to 4imesH100 (DP=2 over TP=2), PEEK delivers up to 3.0imes/2.6imes cache hit, 7.9imes/7.1imes TTFT, 6.7imes/5.5imes E2E, and 3.6imes/4.5imes throughput gains over each engine's strongest stock baseline (SGLang/vLLM), while matching baselines within noise on workloads with no exploitable prefix structure. Wins hold as KV-cache pressure and inference parallelism scale.

Source: arXiv cs.AI | 2026-07-07