Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention

arXiv:2610.02816v1 Announce Type: new Abstract: Linear attention models have emerged as efficient alternatives to standard attention, but effectively managing their fixed-size recurrent memory remains challenging. To improve memory, recent work has explored two distinct directions: delta-rule variants for precise correction of values associated with keys, and slot-based architectures such as Gated Slot Attention for modeling key and value memories in two stages. We observe that these directions are complementary--the delta rule provides effective memory correction, while the two-stage structure provides a natural way to operate on both sides of an association. Building on this insight, we introduce a new Gated Oja Rule for key-side correction and extend it with decoupled erase and write control to obtain Gated Oja Rule-2. We then introduce Gated Slot Attention-2 (GSA2), which combines Gated Oja Rule-2 for key-side correction with Gated Delta Rule-2 for value-side correction through shared latent slots. We further derive a hardware-efficient chunkwise algorithm for parallel training. Experiments demonstrate that GSA2 consistently improves over strong linear-attention baselines across benchmarks while retaining linear-time sequence modeling and constant-memory recurrent decoding.

Related

• A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets
• The Query Knows What to Forget: A Second Erase Direction for Linear Attention
• Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
• How Linear Attention Remembers

Source: arXiv cs.LG | 2026-10-05