Research
Plateaus, Optima, and Overfitting in Multi-Layer Perceptrons: A Saddle-Saddle-Attractor Scenario
arXiv:2604.02393v2 Announce Type: replace Abstract: Vanishing gradients and overfitting are central problems in machine learning, yet are typically analyzed in asymptotic regimes that obscure their dy
arXiv:2604.02393v2 Announce Type: replace Abstract: Vanishing gradients and overfitting are central problems in machine learning, yet are typically analyzed in asymptotic regimes that obscure their dynamical origins. Here we provide a dynamical description of learning in multi-layer perceptrons (MLPs) via a minimal model inspired by Fukumizu and Amari. We show that training dynamics traverse plateau and near-optimal regions, both organized by saddle structures, before converging to an overfitting regime. Under suitable conditions on the data, this regime collapses to a single attractor modulo symmetry. Furthermore, for finite noisy datasets, convergence to the theoretical optimum is impossible, and the dynamics necessarily settle into an overfitting solution.
Related
- Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models
- Mitigating Barren Plateaus in Quantum Denoising Diffusion Probabilistic Model
- Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix
Source: arXiv cs.LG | 2026-04-20