Model Releases
Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
arXiv:2608.13335v1 Announce Type: new Abstract: Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops ab
arXiv:2608.13335v1 Announce Type: new Abstract: Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly. Meanwhile, training losses instead follow smooth power laws. Variants of both behaviors occur in architectures with very different microscopic structures, which is the signature of a few relevant collective variables. We show that a symmetry fixes what those variables are: a network layer is a sum over interchangeable units, so relabeling the units leaves it unchanged; given smoothness and the condition that a unit's gradient vanish at the origin, symmetry then enforces a universal leading form for the expansion about the near-zero weights present at the start of training, the quadratic Tr[WW^{op}A(x)], in which every architectural detail is confined to a single structure matrix" A(x) that we compute for each architecture. Perceptrons, attention layers, mixtures of experts, and convolutions become one model at different A. Its training dynamics then close on the order parameter" M=WW^{op} and, whenever the data matrices share an eigenbasis, reduce to a Lotka--Volterra equation whose modes switch on one after another. The smaller the initial weights, the further apart the switch-on times, and the plateaus appear as a singular limit of a smooth flow; when many modes are unresolved the same events merge into a power law in training time whose exponent the theory predicts. We confirm both numerically across training methods and architectures.
Related
- The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning
- A ghost mechanism: An analytical model of abrupt learning in recurrent networks
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons
- Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness
Source: arXiv cs.LG | 2026-08-14