Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

arXiv:2606.02596v1 Announce Type: new Abstract: The curvature exponent alpha in h_k propto sigma_k^alpha -- governing how Hessian eigenvalues scale with gradient singular values -- varies systematically across layer types (alpha approx 2 for convolutions, approx 1 for transformer attention, < 1 for MLP up-projections). Why? We prove the Spectral Alignment Decomposition: alpha = 2 + dlogPhi_k / dlogsigma_k, where Phi_k measures alignment between Kronecker factor eigenbases and gradient singular directions. This reduces "why does alpha vary?" to a geometric question we answer for LayerNorm, residual connections, and softmax heads. The decomposition implies a spectral transfer identity s = alphagamma linking curvature exponent, effective gradient rank-decay gamma, and Hessian decay exponent s. The identity is algebraic; its empirical content is that alpha and gamma, fit on independent data (HVPs vs. SVD), recover s to ~2% median error across 93 layers, five architectures, and three datasets -- with no free parameters. A zeta-function bound on participation ratio shows curvature concentrates onto effectively one direction per layer. As a proof of concept, we derive the architecture-adaptive preconditioner T(sigma;alpha) and show that Spectral Newton -- implementing T in the gradient singular basis -- outperforms AdamW on vision benchmarks where alpha approx 2.

Source: arXiv cs.LG | 2026-06-03