Tutorials
Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothness
arXiv:2609.30276v1 Announce Type: new Abstract: We analyze the original same-step coordinate-wise AdaGrad under generalized smoothness and heavy-tailed noise with bounded variance. In this setting, lo
arXiv:2609.30276v1 Announce Type: new Abstract: We analyze the original same-step coordinate-wise AdaGrad under generalized smoothness and heavy-tailed noise with bounded variance. In this setting, local curvature may grow sub-quadratically with the gradient norm, and stochastic gradients are assumed to have only bounded conditional second moments. We show that unclipped AdaGrad can become anisotropically miscalibrated: under heavy-tailed noise, the adaptive denominator can learn the geometry of rare noise shocks rather than the local curvature of the objective, leading to a persistent directional distortion that blocks finite-horizon Euclidean progress. We then prove that clipping repairs this failure mode. Our main result is a finite-horizon high-probability guarantee for the original non-lagged AdaGrad update, yielding frac1Tsum_{t=0}^{T-1}|nabla f(x_t)|^2=Oleft(frac{dig(sqrt{log T} + log frac{1}{elta}ig)}{sqrt{T}}right), and hence widetilde{mathcal O}(arepsilon^{-2}) complexity. This shows that, for AdaGrad under heavy-tailed noise, clipping is a structural stabilizer of the adaptive geometry rather than merely a robustness heuristic.
Related
Source: arXiv cs.LG | 2026-09-28