Representational Curvature Modulates Behavioral Uncertainty in Large Language Models

arXiv:2604.23985v1 Announce Type: new Abstract: In autoregressive large language models (LLMs), temporal straightening offers an account of how the next-token prediction objective shapes representations. Models learn to progressively straighten the representational trajectory of input sequences across layers, potentially facilitating next-token prediction via linear extrapolation. However, a direct link between this trajectory and token-level behavior has been missing. We provide such a link by relating contextual curvature-a geometric measure of how sharply the representational trajectory bends over recent context-to next-token entropy. Across two models (GPT-2 XL and Pythia-2.8B), contextual curvature is correlated with entropy, and this relationship emerges during training. Perturbation experiments reveal selective dependence: manipulating curvature through trajectory-aligned interventions reliably modulates entropy, while geometrically misaligned perturbations have no effect. Finally, regularizing representations to be straighter during training modestly reduces token-level entropy without degrading validation loss. These results identify trajectory curvature as a task-aligned representational feature that influences behavioral uncertainty in LLMs.

Related

• Learning and Enforcing Context-Sensitive Control for LLMs
• Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards
• Language Models Learn Universal Representations of Numbers and Here's Why You Should Care
• Transformers Can Learn Connectivity in Some Graphs but Not Others
• A Mechanistic Analysis of Looped Reasoning Language Models

Source: arXiv cs.AI | 2026-04-28