Revisiting the Uniform Information Density Hypothesis in LLM Reasoning

arXiv:2510.06953v3 Announce Type: replace Abstract: The Uniform Information Density (UID) hypothesis proposes that effective communication is achieved by maintaining a stable flow of information. In this work, we revisit this principle in the context of Large Language Model (LLM) reasoning, asking whether step-level uniformity reflects reasoning quality. To this end, we introduce a novel framework to quantify uniformity of information flow at both local and global levels, using an entropy-based stepwise density metric. Across experiments on seven reasoning benchmarks, we see a counter-intuitive pattern: while high-quality reasoning exhibit smooth step-by-step transitions local uniformity and structured, non-uniform information flow at the trajectory level global non-uniformity. The results demonstrate that these uniformities outperform alternative internal signals as predictors of reasoning quality, and such divergence with human communication is not a model deficiency, but a byproduct of distinct objectives between human communication and LLM reasoning.

Related

• Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution
• Process Reward Agents for Steering Knowledge-Intensive Reasoning
• HintMR: Eliciting Stronger Mathematical Reasoning in Small Language Models
• Revitalizing Black-Box Interpretability: Actionable Interpretability for LLMs via Proxy Models

Source: arXiv cs.AI | 2026-04-20