Last-Iterate Convergence Rate of Normalized Gradient Descent under Holder Smoothness

arXiv:2610.06070v1 Announce Type: cross Abstract: Normalized gradient descent is a widely studied adaptive optimization method. Most existing analyses focus on the best iterate or a weighted average of the iterates, whereas practical implementations typically return the last iterate. In this paper, we study the last-iterate convergence of normalized gradient descent for convex, (nu,M_nu)-Holder-smooth objectives. For a constant stepsize, we establish an upper bound of Oigl((log^2(T)/T)^{(1+nu)/2}igr), which contains a logarithmic overhead relative to the known Oigl(T^{-(1+nu)/2}igr) guarantees for the best and weighted-average iterates. For nu = 0, this overhead is known to be unavoidable. We complement this analysis with numerical results based on the performance estimation problem (PEP), investigating the finite-horizon worst-case behavior in the smooth setting and whether the logarithmic overhead reflects an intrinsic limitation of constant-step normalized gradient descent. We then show that a linearly decreasing stepsize yields a last-iterate guarantee of Oigl(T^{-(1+nu)/2}igr), matching the order of the best-iterate/weighted-average guarantees without requiring knowledge of nu and M_nu.

Related

• A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity
• Gradient Descent's Last Iterate is Often (slightly) Suboptimal
• Revisiting AdaGrad in Stochastic Convex Optimization: Last Iterates, High Probability, and Lower Bounds
• ZENITH: Automated Gradient Norm Informed Stochastic Optimization

Source: arXiv cs.LG | 2026-10-06