SignMuon: Communication-Efficient Distributed Muon Optimization

arXiv:2605.16311v1 Announce Type: new Abstract: Distributed training of large neural networks is bottlenecked by full-precision gradient communication and by coordinatewise optimizers that ignore the matrix structure of weight tensors. We propose Sign-Muon, a 1-bit, matrix-aware optimizer that combines majority-vote sign aggregation from signSGD with the polar-step framework of Muon. Each worker forms a Muon-style direction by taking the polar factor of its momentum via a Newton--Schulz iteration, transmits only the entrywise signs, and aggregates by majority vote; an optional local polar step further enforces orthogonality at no extra communication cost. Under spectral-norm smoothness and bounded-variance stochastic gradients, the spectral-norm normalized sign step yields an O(1/sqrt{T}) nonconvex rate for an ell_1-based stationarity measure. With unimodal symmetric noise, majority vote across M workers cuts the stochastic term by 1/sqrt{M}, matching signSGD. In the alpha-eta model, distributed Sign-Muon needs only one integer sum-allreduce per iteration; all orthogonalization is local, giving a 32imes bandwidth reduction over float32 (4imes for int8). Across 330 CIFAR-10/ResNet-50 configurations Sign-Muon attains the best validation accuracy (92.15%); its 4-GPU majority-vote variant reaches 92.02% with 37% less training time at matched effective batch. On nanoGPT, Sign-Muon achieves lower perplexity and better anytime performance than other sign-based baselines, with favorable weak-scaling up to 16 GPUs.

Source: arXiv cs.LG | 2026-05-19