Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
DGX agentarXiv:2605.17787v1 Announce Type: new Abstract: It is widely believed that stochastic gradient descent (SGD) performs significantly worse than adaptive optimizers such as Adam in pre-training Large La