Research
RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States
arXiv:2609.12814v1 Announce Type: new Abstract: Linear attention and state-space models provide linear-time sequence modeling, but their recurrent memory remains a second-order tensor (a matrix), limi
arXiv:2609.12814v1 Announce Type: new Abstract: Linear attention and state-space models provide linear-time sequence modeling, but their recurrent memory remains a second-order tensor (a matrix), limiting the order of interactions that can be represented in the state. We introduce the RunningTensor, which generalizes this memory to an order-o tensor, updated by a rank-1 outer product and read by contracting against o-1 vector queries. Order 2 recovers linear attention; we study order 3 as a proof of concept, retaining both recurrent and parallel forms while remaining linear in sequence length T and improving working memory capacity from O(W^2) to O(W^o). On synthetic multi-query associative recall, RunningTensor outperforms linear-attention and SSM baselines. After pretraining, it also improves performance on language-understanding and non-synthetic retrieval tasks, suggesting that higher-order recurrent state can provide useful additional memory capacity beyond matrix-valued state.
Related
- DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
- Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
- Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels
- Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modeling
Source: arXiv cs.LG | 2026-09-14