Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

arXiv:2610.13708v1 Announce Type: new Abstract: Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not handle draft-model speculative decoding such as DSpark and DFlash2. We present Adaptive DHiSparse, which supports speculative decoding over hierarchical sparse KV and adapts residency to load. It protects the required working set, retains additional history while GPU memory is spare, and recovers the rest through exact fused retrieval. DSpark and DFlash2 keep their proposal and acceptance rules: queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45imes HiSparse and 1.27imes non-speculative adaptive serving. Held to the same latency bounds as HiSparse, it sustains up to 4imes the concurrency and 2.3 to 3.7imes the output, depending on the per-token bound. On DeepSeek-V4.1-Flash at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85imes the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2% to 1.4% of selected rows at low load and releases memory under pressure.

Related

• AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD
• IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
• S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices
• Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes

Source: arXiv cs.AI | 2026-10-12