On the Convergence of Adam, Revisited

arXiv:2607.03519v1 Announce Type: new Abstract: We show that projected Adam for online optimization with arbitrary moment decay parameters eta_1,eta_2in[0,1) can have average regret bounded away from zero. A similar result of Reddi-Kale-Kumar from 2018 required eta_1<sqrt{eta_2}. Similar to their result, we use a three-periodic sequence of linear functions on [-1,1] with slopes c,-1,-1, though we use c slightly larger than 2. This nonzero average regret result extends to Adam variants such as AdamW, RMSProp, NAdam, Adan, AdaMax, Muon, and to an i.i.d. variant of the three-periodic sequence of slopes for Adam.

Source: arXiv cs.LG | 2026-07-07