On the Convergence of Stochastic Low-Rank Adaptation

arXiv:2607.21975v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) optimizes J(B,A)=mathcal L(W_base+sBA) over two adapters B in R^{m imes r} and A in R^{r imes n} that form a low-rank update to a frozen pretrained weight matrix W_base in R^{m imes n}. The prior analysis shows LoRA-GD takes exp{O(epsilon^{-2})} oracle calls to find an epsilon-stationary point such that |nabla J(B,A)|leq epsilon in the deterministic setting. We sharpen the analysis and show that O(epsilon^{-4}) full-gradient evaluations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased gradient estimates and finite variance. We propose LoRA-NSGDM, which finds an epsilon-stationary point with O(epsilon^{-8}) stochastic oracle complexity. Under the additional mean-square smoothness condition, we use variance reduction strategy and propose LoRA-STORM, which improves the stochastic oracle complexity to O(epsilon^{-6}).

Related

• On the Convergence Rate of LoRA Gradient Descent
• Last-Iterate Convergence of Randomized Kaczmarz and SGD with Greedy Step Size
• PLoRA: Efficient Concurrent LoRA Training for Large Language Models

Source: arXiv cs.LG | 2026-07-27