SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
arXiv:2607.20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges hav