Spectral Entropy Collapse as an Empirical Signature of Delayed Generalisation in Grokking

arXiv:2604.13123v1 Announce Type: new Abstract: Grokking -- delayed generalisation long after memorisation -- lacks a predictive mechanistic explanation. We identify the normalised spectral entropy ilde{H}(t) of the representation covariance as a scalar order parameter for this transition, validated on 1-layer Transformers on group-theoretic tasks. Five contributions: (i) Grokking follows a two-phase pattern: norm expansion then entropy collapse. (ii) ilde{H} crosses a stable threshold ilde{H}^* approx 0.61 before generalisation in 100% of runs (mean lead: 1,020 steps). (iii) A causal intervention preventing collapse delays grokking by +5,020 steps (p=0.044); a norm-matched control (n=30, p=5imes10^{-5}) confirms entropy -- not norm -- drives the transition. (iv) A power-law Delta T = C_1(ilde{H}-ilde{H}^*)^gamma+C_2 (R^2=0.543) predicts grokking onset with 4.1% error. (v) The mechanism holds across abelian (Z/97Z) and non-abelian (S_5) groups. Crucially, MLPs show entropy collapse without grokking, proving collapse is necessary but not sufficient -- architecture matters. Code: https://anonymous.4open.science/r/grokking-entropy

Source: arXiv cs.LG | 2026-04-16