Sessa: Selective State Space Attention

arXiv:2604.18580v1 Announce Type: cross Abstract: Modern sequence models are dominated by Transformers, where self-attention mixes information from the visible context in an input-dependent way. However, when retrieval is not sharp and attention remains diffuse over an effective support S_{eff}(t), the influence of any individual token is diluted, typically scaling as O(1/S_{eff}(t)) and reaching O(1/ell) for old tokens in full-prefix settings. Structured state-space models process sequences recurrently through an explicit feedback path; selective variants such as Mamba make this feedback input-dependent, yet when freeze time cannot be sustained over long intervals, their long-range sensitivity decays exponentially with lag. Existing architectures therefore either retrieve from the past in a single read or propagate information through a single feedback chain. We introduce Sessa, a decoder that places attention inside a feedback path, enabling recurrent many-path aggregation within a layer. Under stated assumptions, Sessa admits regimes with a power-law memory tail in lag ell of order O(ell^{-eta}) for 0<eta<1, which is asymptotically slower than 1/ell; moreover, this rate is tight in an explicit diffuse uniform-routing setting where the influence is Theta(ell^{-eta}). Under the same conditions, only Sessa among the compared model classes realizes flexible selective retrieval, including non-decaying profiles. Empirically, under matched architectures and training budgets, Sessa achieves the strongest performance on our long-context benchmarks while remaining competitive with Transformer and Mamba style baselines on short-context language modeling.

Source: arXiv cs.CL | 2026-04-21

Related