Kinetic theory for Transformers and the lost-in-the-middle phenomenon

arXiv:2605.09213v1 Announce Type: cross Abstract: We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed lost-in-the-middle phenomenon: the token retrieval profile, as a function of the source position in the prompt, is mathsf{U}-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition.

Source: arXiv cs.LG | 2026-05-12