Model Releases
Nice paper to improve inference efficiency. It's been a while we haven't seen good work on efficiency. Here is why it matters: A long-runnin…
Nice paper to improve inference efficiency. It's been a while we haven't seen good work on efficiency. Here is why it matters: A long-running agent's workspace outgrows its context window long before
Nice paper to improve inference efficiency. It's been a while we haven't seen good work on efficiency. Here is why it matters: A long-running agent's workspace outgrows its context window long before the task finishes. The first approach commonly used, compaction, loses the fine-grained execution evidence. And text retrieval re-prefills content the model already processed. KVMem keeps the overflow as paged KV state instead, spread across GPU memory, host memory and NVMe. Lightweight attention-space indexes, native to the model, pick the relevant historical blocks and materialize a query-dependent view that fits inside the native context window. On the DeepSWE long-context test with Qwen3.8-27B, task success goes from 43.8% under compaction to 48.4%. The local deployment result stands out. It runs Qwen3.6/3.8-27B NVFP4 with MTP on a laptop with a 24GB RTX 5090, virtualizing an agent workspace up to 1M tokens, four times the model's native 256K window, at around 50 tokens per second. Paper: https://academy.dair.ai/papers/kvmem-virtualizing-million-token-agent-workspaces-on-a-consumer-gpu-2609.04852
Related
- Banger paper from NVIDIA. Agentic reasoning needs models that are not just capable, but efficient at long-context inference. The agent model…
- // The Efficiency Frontier // Cool paper on context management. As agents reuse the same documents and histories across many turns, the chea…
- 'Ghost memory' is a real problem with agents. You might have seen the issue where a long-running agent still confidently repeats a user fact…
Source: DAIR.AI (X) | 2026-09-09