Research
Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks
arXiv:2604.09412v1 Announce Type: cross Abstract: We study the population loss landscape of two-layer ReLU networks of the form sum_{k=1}^K ReLU(w_k^op x) in a realisable teacher-student setting with
arXiv:2604.09412v1 Announce Type: cross Abstract: We study the population loss landscape of two-layer ReLU networks of the form sum_{k=1}^K ReLU(w_k^op x) in a realisable teacher-student setting with Gaussian covariates. We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape. We further establish a direct link with one-pass SGD: local minima correspond to attractive fixed points of the dynamics in summary statistics space. This perspective reveals a hierarchical structure of minima: they are typically isolated in the well-specified regime, but become connected by flat directions as network width increases. In this overparameterised regime, global minima become increasingly accessible, attracting the dynamics and reducing convergence to spurious solutions. Overall, our results reveal intrinsic limitations of common simplifying assumptions, which may miss essential features of the loss landscape even in minimal neural network models.
Related
- Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks
- Practical Bayesian Inference for Speech SNNs: Uncertainty and Loss-Landscape Smoothing
- PAC-Bayesian Bounds on Constrained f-Entropic Risk Measures
- ECLipsE-Gen-Local: Efficient Compositional Local Lipschitz Estimates for Deep Neural Networks
Source: arXiv cs.LG | 2026-04-13