The Geometry of Language Model Logits: Generic Structure and Learned Departures

arXiv:2610.18520v1 Announce Type: cross Abstract: Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-token distribution. We empirically identify that the ordered vocabulary-head logits follow a remarkably consistent log-linear relation [ z_{(r)} approx mu - alpha log r. ] across prediction contexts, architectures, and scales, emerging progressively during pretraining. To our knowledge, this is the first systematic analysis of the logit geometry underlying the distributional assumptions used in the prior decoding work. While controlled order-statistic baselines spanning bounded, light-tailed, and heavier-tailed distributions can largely explain the observed log-linear geometry, they do not explain the systematic shift in the slope alpha: trained language models exhibit alpha approx 1.4--1.7, sitting well above the initialization baseline (alpha approx 0.2) . Moreover, alpha correlates with predictive performance, suggesting that it captures learned structure beyond generic extreme-value effects.

Related

• Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens
• Gumbel Straight Flow: Distilling Autoregressive Models into One-step Flow Maps
• Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement
• Harmonizing Multi-Objective LLM Unlearning via Unified Domain Representation and Bidirectional Logit Distillation
• Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization

Source: arXiv cs.AI | 2026-10-15