Research
The Depth Flow of Token Representations Is Nonlinear and Does Not Descend Its Own Density
arXiv:2608.29706v1 Announce Type: new Abstract: A token's representation is carried through the network layer by layer. The whole vocabulary carried together forms a flow. We fit this flow's equation
arXiv:2608.29706v1 Announce Type: new Abstract: A token's representation is carried through the network layer by layer. The whole vocabulary carried together forms a flow. We fit this flow's equation of motion as a discrete Langevin model over corpus-mean trajectories of Pythia-160M and Pythia-410M, and score the predicted steps on held-out tokens. Linear maps are often used as cheap surrogates for a layer. The flow they summarize is not linear: a quadratic drift beats the linear linear map at every transition of both models, and the Kramers--Moyal estimator agrees wherever its neighborhoods stay local. We then characterize the flow further. First, we show that it does not descend its own log-density. The drift instead descends a potential that is not the density. Second, the rotational component is not negligible, 4 to 45% of the explainable drift, and the circulation shows in what the flow preserves: a token keeps its angular rank across all thirteen layers while its norm rank is shuffled and its concentration rank is reversed by the last block.
Related
- WhiteMatter: All-to-All Cross-Layer Connections via KV Mixing
- TANGO: Token-Aggregated Nonlinear Gating Operators for Natural and Formal Language Modeling
- How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs
- Semantic Density Effect (SDE): Maximizing Information Per Token Improves LLM Accuracy
Source: arXiv cs.CL | 2026-09-01