When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

arXiv:2610.21555v1 Announce Type: cross Abstract: Muon's gains in data efficiency motivate evaluating its performance under explicit data, time, and tuning budgets. On a single A100 GPU, we compare a reference single-device Muon implementation with AdamW across 17.7M- and 33.6M-parameter causal language models and 2.70M-parameter vision transformers. A common 23-candidate development budget selects configurations evaluated with five matched seeds, full validation trajectories, optimizer-step timing, and a frozen one-shot test. Muon reduces validation-loss area under the curve by 3.25--22.44% at equal data and lowers test loss in all 20 primary pairs. In this configuration, Muon improves time-normalized loss area} by 19.68--22.31% on vision, while AdamW is better by 3.38--6.96% on language. The full Muon optimizer call, including auxiliary AdamW updates, consumes 56.1--69.6% of training time, compared with 6.1--9.1% for the AdamW baseline. Muon also has higher expected regret under restricted access to these candidate pools. These results characterize small models with reference kernels; their wall-clock ordering does not establish a ranking for larger models or optimized implementations.

Related

• Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency
• Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
• GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgets

Source: arXiv cs.AI | 2026-10-16