Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stability

arXiv:2604.02653v2 Announce Type: replace Abstract: Empirically, modern deep learning training often occurs at the Edge of Stability (EoS), where the sharpness of the loss exceeds the threshold below which classical convergence analysis applies. Despite recent progress, existing theoretical explanations of EoS either rely on restrictive assumptions or focus on specific squared-loss-type objectives. In this work, we introduce and study a structural property of loss functions that we term product-stability. We show that for losses with product-stable minima, gradient descent applied to objectives of the form (x,y) mapsto l(xy) can provably converge to the local minimum even when training in the EoS regime. This framework substantially generalizes prior results and applies to a broad class of losses, including binary cross entropy. Using bifurcation diagrams, we characterize the resulting training dynamics, explain the emergence of stable oscillations, and precisely quantify the sharpness at convergence. Together, our results offer a principled explanation for stable EoS training for a wider class of loss functions.

Related

• Large Spikes in Stochastic Gradient Descent: A Large-Deviations View
• Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks
• Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification
• Fisher-Geometric Diffusion in Stochastic Gradient Descent: Optimal Rates, Oracle Complexity, and Information-Theoretic Limits

Source: arXiv cs.LG | 2026-04-14