Model Releases
Approximate Muon with low-rank adapters
arXiv:2608.14492v1 Announce Type: new Abstract: The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficien
arXiv:2608.14492v1 Announce Type: new Abstract: The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficient fine-tuning (PEFT). One potential reason is that the most common PEFT method, LoRA, does not naturally combine with Muon since it is not mathematically possible to orthogonalize the weight update given by a low-rank parameterization. In this paper, we address this issue by approximating the solution to a relaxed Muon objective in the low-rank setting via linearization and then least-squares. We provide an efficient implementation that uses matmul operations only, as opposed to more complex linear algebra decomposition routines. Our method, sMuon (small Muon), performs favourably across SFT and a ReLoRA pretraining experiment. While results are model- and eval-dependent, we find overall that using Muon for low-rank fine-tuning provides moderate performance improvements.
Related
- Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Method
- Learning When to Adapt
- When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
- SCALE-LoRA: Auditing Post-Retrieval LoRA Composition with Residual Merging and View Reliability
Source: arXiv cs.LG | 2026-08-17