New research from NVIDIA. Does AdamW have a scale ceiling? This work claims yes, and shows where it sits. At batch sizes up to 100M tokens f…
DGX agentNew research from NVIDIA. Does AdamW have a scale ceiling? This work claims yes, and shows where it sits. At batch sizes up to 100M tokens for next-token prediction, SOAP and Muon maintain training st