Research
Asymptotic Behavior of Transformers
arXiv:2610.19358v1 Announce Type: new Abstract: In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer att
arXiv:2610.19358v1 Announce Type: new Abstract: In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer attention-only transformer can fail to converge. We then study a rich class of transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers. Every transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions. We show further that, while each transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all transformers to converge.
Related
- Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers
- DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
- Continual Learning Mechanisms Compose for Long-Horizon Memorization
Source: arXiv cs.LG | 2026-10-15