Model Releases
focus-llama: a llama.cpp fork implementing Declarative Attention (arXiv:2609.02737)
I forked llama.cpp to implement Declarative Attention (arXiv:2609.02737, Google DeepMind and KAIST AI). The model declares in its own output which context chunks it needs <focus magic_chunks='N'>), an
I forked llama.cpp to implement Declarative Attention (arXiv:2609.02737, Google DeepMind and KAIST AI). The model declares in its own output which context chunks it needs ), and the engine listens and restricts what the following tokens can attend to. No scorer, no training: just prompting plus an engine that reacts to the tags. According to their paper, the overall decode time for the response can be reduced to 0.71× (Gemma) and 0.77× (Qwen) compared to vanilla. Those are the paper's vLLM numbers; I haven't benchmarked the fork yet (no accuracy benchmark either). = What works = - llama-server can drop KV token ranges for a request da_rm / da_rm_at), either mid-prefill or after prefill. - Tag-driven mode: given a chunk layout from the client, the server parses the model's first tag during generation and drops the other chunks at that point (one-shot, one-way). - A second-sequence mode da_b, needs --kv-unified) leaves the original sequence intact. - Checked with first-token logprobs on small smoke tests (a Qwen hybrid/GDN model and a tiny dense one). = Why a fork = - Stock llama-server has no way to touch KV ranges mid-generation, so this needed server changes. - llama.cpp has no paged block table (the paper's vLLM version rewrites it), so masking alone doesn't reduce the KV that gets read. - Real skipping needs kernel support or compaction. From reading the source, single-token decode on CUDA doesn't skip masked chunks today. - On hybrid models only the attention layers are restricted (same as the paper); the recurrent state is untouched. Repo: https://github.com/edwardyoon/focus-llama Feedback welcome, especially from anyone who has worked on llama.cpp KV/recurrent memory. submitted by /u/Ok-Shower7286 [link] [comments]
Related
- Update: llama.cpp for Radeon VII / MI50 / MI60 — +14% PP, +9% long-context fill vs upstream + adaptive Flash Attention
- Llama.cpp DSpark PC Tree Fork (up to 3%-29.5% faster!)
- CachyLLama’s: llama.cpp fork with persistent KV cache that makes long local-agent sessions much less painful
Source: r/LocalLLaMA | 2026-09-20