RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State

arXiv:2602.11958v2 Announce Type: replace-cross Abstract: Linear attention offers an efficient alternative to full attention with a fixed-size recurrent state. However, this state is shared by all tokens, so information from distinct tokens becomes superposed within it and produces inter-token interference that degrades long-range fine-grained recall. To address this issue, we propose RAM-Net, which replaces dense access to a shared state with sparse address-based access. RAM-Net organizes the recurrent state as a fixed-size array of independent slots and uses an Address Decoder that maps each key or query into a sparse address, selecting a small subset of slots to write to or read from at each step. This design directs tokens with non-overlapping addresses to disjoint slots, suppressing inter-token interference, while keeping per-step state access dependent only on the number of selected slots rather than the total state size. Empirically, RAM-Net outperforms strong recurrent baselines on fine-grained long-range retrieval and achieves the lowest perplexity with competitive commonsense reasoning. It does so while accessing fewer state elements per step than all baselines, e.g., 8imes fewer than Mamba2.

Related

• What Attention Recalls and Recurrence Controls in Hybrid Language Models
• A Systematic Analysis of Hybrid Linear Attention
• On State Reduction in Linear Attention
• Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
• RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States

Source: arXiv cs.CL | 2026-10-07