Model Releases
From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias
arXiv:2610.20112v1 Announce Type: new Abstract: Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (2
arXiv:2610.20112v1 Announce Type: new Abstract: Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (2025) established spectral max-margin behavior for spectral-norm steepest descent methods, including idealized Muon with exact orthogonalization. In practice, however, Muon approximates orthogonalization using finite Newton-Schulz iterations and a positive stabilization constant. We prove that these implementation choices cause Muon to select the Frobenius rather than the spectral max-margin direction in linear classification with separable data. We also establish this behavior for both exponential moving average and Nesterov-style momentum. The key technical ingredient is a summable perturbation argument showing that the resulting parameter trajectories remain at bounded distance from those of gradient descent. We then disentangle the roles of Newton-Schulz iteration count, the stabilization constant, and the Newton-Schulz polynomial, revealing distinct asymptotic behaviors, including directions that achieve neither Frobenius nor spectral max-margin. Finally, we show that Muon can recover its spectral max-margin bias with Newton-Schulz iterations growing logarithmically over training.
Related
- LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates
- Approximate Muon with low-rank adapters
- When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging
- AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Source: arXiv cs.LG | 2026-10-15