Momentum Further Constrains Sharpness at the Edge of Stochastic Stability

arXiv:2604.14108v1 Announce Type: new Abstract: Recent work suggests that (stochastic) gradient descent self-organizes near an instability boundary, shaping both optimization and the solutions found. Momentum and mini-batch gradients are widely used in practical deep learning optimization, but it remains unclear whether they operate in a comparable regime of instability. We demonstrate that SGD with momentum exhibits an Edge of Stochastic Stability (EoSS)-like regime with batch-size-dependent behavior that cannot be explained by a single momentum-adjusted stability threshold. Batch Sharpness (the expected directional mini-batch curvature) stabilizes in two distinct regimes: at small batch sizes it converges to a lower plateau 2(1-eta)/eta, reflecting amplification of stochastic fluctuations by momentum and favoring flatter regions than vanilla SGD; at large batch sizes it converges to a higher plateau 2(1+eta)/eta, where momentum recovers its classical stabilizing effect and favors sharper regions consistent with full-batch dynamics. We further show that this aligns with linear stability thresholds and discuss the implications for hyperparameter tuning and coupling.

Related

• Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stability
• Large Spikes in Stochastic Gradient Descent: A Large-Deviations View
• Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks
• Fisher-Geometric Diffusion in Stochastic Gradient Descent: Optimal Rates, Oracle Complexity, and Information-Theoretic Limits
• Classical and Quantum Speedups for Non-Convex Optimization via Energy Conserving Descent

Source: arXiv cs.LG | 2026-04-16