Banger paper from Google DeepMind and colleagues. (bookmark it) A model reads its entire KV cache on every generated token, even though it e…

Banger paper from Google DeepMind and colleagues. (bookmark it) A model reads its entire KV cache on every generated token, even though it ends up attending to a tiny slice of it. In other words, if you ask about one detail from a 1M-token conversation the global attention layers re-read all of it, per token. The usual fix is to guess the relevant tokens first with cheap proxy scores, which still costs O(N) every step. Declarative Attention asks the model instead. The model declares where it needs to look, inside its own chain-of-thought. In this way, generation splits into three modes: global reads the full context, focus reads one specific region, and local reads only recent output. The inference engine parses those declarations the same way it parses tool calls and skips most of the cache read. On zero-shot on off-the-shelf weights across 15 long-context tasks, attended tokens during decoding drop 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B. Paper: https://arxiv.org/abs/2609.02737 Chat with Paper: https://academy.dair.ai/papers/language-models-can-control-their-own-attention-2609.02737

Source: DAIR.AI (X) | 2026-09-03

Related