Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization
DGX agentarXiv:2502.00213v5 Announce Type: replace-cross Abstract: Transformers are difficult to optimize with stochastic gradient descent (SGD) and largely rely on adaptive optimizers such as Adam. Despite ex