OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality

arXiv:2603.09923v3 Announce Type: replace Abstract: The Exponential Moving Average (EMA) is a cornerstone of widely used optimizers such as Adam. However, existing theoretical analyses of Adam-style methods have notable limitations: their guarantees can remain suboptimal in the zero-noise regime, rely on restrictive boundedness conditions (e.g., bounded gradients or objective gaps), use constant or open-loop stepsizes, or require prior knowledge of Lipschitz constants. To overcome these bottlenecks, we introduce OptEMA and analyze two novel variants: OptEMA-M, which applies an adaptive, decreasing EMA coefficient to the first-order moment with a fixed second-order decay, and OptEMA-V, which swaps these roles. At the heart of these variants is a novel Corrected AdaGrad-Norm stepsize. This formulation renders OptEMA closed-loop and Lipschitz-free, meaning its effective stepsizes are strictly trajectory-dependent and require no parameterization via the Lipschitz constant. Under standard stochastic gradient descent (SGD) assumptions, namely smoothness, a lower-bounded objective, and unbiased gradients with bounded variance, we establish rigorous convergence guarantees. Both variants achieve a noise-adaptive convergence rate of widetilde{O}(T^{-1/2}+sigma^{1/2} T^{-1/4}) for the average gradient norm, where sigma is the noise level. Crucially, the Corrected AdaGrad-Norm stepsize plays a central role in enabling the noise-adaptive guarantees: in the zero-noise regime (sigma=0), our bounds automatically reduce to the nearly optimal deterministic rate widetilde{O}(T^{-1/2}) without any manual hyperparameter retuning.

Related

• DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling
• High Probability Guarantees for Random Reshuffling
• Fisher-Geometric Diffusion in Stochastic Gradient Descent: Optimal Rates, Oracle Complexity, and Information-Theoretic Limits
• Non-Expansive Mappings in Two-Time-Scale Stochastic Approximation: Finite-Time Analysis
• Momentum Further Constrains Sharpness at the Edge of Stochastic Stability
• Large Spikes in Stochastic Gradient Descent: A Large-Deviations View

Source: arXiv cs.LG | 2026-04-17