Dimensional Criticality at Grokking Across MLPs and Transformers

arXiv:2604.16431v1 Announce Type: new Abstract: Abrupt transitions between distinct dynamical regimes are a hallmark of complex systems. Grokking in deep neural networks provides a striking example -- an abrupt transition from memorization to generalization long after training accuracy saturates -- yet robust macroscopic signatures of this transition remain elusive. Here we introduce extbf{TDU--OFC} (Thresholded Diffusion Update--Olami-Feder-Christensen), an offline avalanche probe that converts gradient snapshots into cascade statistics and extracts a macroscopic observable -- the time-resolved effective cascade dimension D(t) -- via grokking-aligned finite-size scaling. Across Transformers trained on modular addition and MLPs trained on XOR, we discover a localized dynamical crossing of the Gaussian diffusion baseline D=1 precisely at the generalization transition. The crossing direction is task-dependent: modular addition descends through D=1 (approaching from D>1), while XOR ascends (from D1) and never enter the post-transition regime. In addition, avalanche distributions exhibit heavy tails and finite-size scaling consistent with the dimensional exponent extracted from D(t). Shadow-probe controls (alpha_{train}=0) confirm that D(t) is non-invasive, and grokked trajectories diverge from ungrokked ones in D(t) some 100--200 epochs before the behavioral transition.

Related

• Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models
• Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix
• Distinct mechanisms underlying in-context learning in transformers
• Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks
• An overview of condensation phenomenon in deep learning

Source: arXiv cs.LG | 2026-04-21