On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature

arXiv:2602.05600v2 Announce Type: replace Abstract: Stochastic Gradient Descent (SGD) introduces anisotropic noise that is correlated with the local curvature of the loss landscape, thereby biasing optimization toward flat minima. Prior work often assumes an equivalence between the Fisher Information Matrix and the Hessian for negative log-likelihood losses, leading to the claim that the SGD noise covariance mathbf{C} is proportional to the Hessian mathbf{H}. We show that this assumption holds only under restrictive conditions that are typically violated in deep neural networks. Using the recently discovered Activity--Weight Duality, we find a more general relationship agnostic to the specific loss formulation, showing that mathbf{C} propto E_p[mathbf{h}_p^2], where mathbf{h}p denotes the per-sample Hessian with mathbf{H} = E_p[mathbf{h}p]. As a consequence, mathbf{C} and mathbf{H} commute approximately rather than coincide exactly. We further find that, within the analyzed fully connected layers, their diagonal elements follow per-layer empirical power laws C{ii} propto H{ii}^{gamma}, with layer-dependent fitted exponents bounded by 1 leq gamma leq 2. Experiments across datasets, architectures, and loss functions support the resulting layerwise bounds, providing a unified characterization of the noise-curvature relationship in deep learning.

Source: arXiv cs.LG | 2026-06-09