MuCon: Clipped Muon Updates for LLM Training

arXiv:2605.26459v1 Announce Type: new Abstract: Muon-style optimizers take a matrix-valued momentum or preconditioned update B = U operatorname{diag}(sigma_1,ldots,sigma_r) V^op and replace it with its canonical partial polar factor operatorname{Pol}(B) = U V^op. This maps every nonzero singular value to one. MuCon is the clipped-Muon variant studied here: it applies singular-value clipping to the same Muon matrix, D^{MuCon}_au(B) = operatorname{MClip}_au(B) = U operatorname{diag}igl(min{sigma_i,au}igr) V^op, qquad au > 0. Thus, operatorname{MClip}_au denotes the mathematical clipping operator, while MuCon denotes the optimizer primitive that substitutes this clipped direction for Muon's polar direction. The Muon/MuCon scaling parameterization used in this work is called ext{SpectralP}: it is the hidden-matrix scaling recipe under which polar Muon or clipped MuCon directions are applied. The map operatorname{MClip}_au is the Frobenius projection onto the spectral-norm ball {X : |X|_2 le au}: it leaves singular values at or below au unchanged and modifies only the violating singular directions. This paper asks when the MuCon clipping step can be approximated without a full dense SVD. We record two exact identities, a polar/absolute-value formula and a scalar-root formulation leading to a rational Newton filter for the clipped positive-semidefinite factor, and identify the numerical obstruction common to both: singular values near the threshold make sign decisions and rational solves ill-conditioned. Matrix-function methods are therefore useful only when paired with stable polar/square-root primitives or explicit regularization near the clipping boundary.

Source: arXiv cs.LG | 2026-05-27