AdamX: Cosine similarity meets gradient descent

arXiv:2609.11867v1 Announce Type: new Abstract: We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training. Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures. Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget. Code and Experiments available at: https://github.com/FranciscoCaldas/adamX.

Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
SAGE: Sign-Adaptive Gradient for Memory-Efficient LLM Optimization
Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
Convergence of Steepest Descent and Adam under Non-Uniform Smoothness
POP: Prior-Fitted First-Order Optimization Policies

Source: arXiv cs.LG | 2026-09-11

Related