Model Releases
Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces
arXiv:2610.15225v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs). However, ZO optimization suffers from high variance
arXiv:2610.15225v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs). However, ZO optimization suffers from high variance in high-dimensional parameter spaces. While recent methods restrict perturbations to random low-rank subspaces, these subspaces often misalign with task-specific descent directions. We propose AdaSub-ZO, a novel forward-only fine-tuning framework in learnable low-rank subspaces. Instead of relying on fixed random matrices, AdaSub-ZO iteratively learns layer-wise projection matrices directly from finite-difference feedback to track task-aware descent directions. Furthermore, AdaSub-ZO incorporates a subspace momentum acceleration mechanism to smooth noisy gradient estimates across evolving subspace bases. Extensive experiments on natural language understanding tasks demonstrate that AdaSub-ZO outperforms existing ZO baselines, substantially reducing gradient estimation variance and accelerating convergence.
Related
- ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
- SubZero+: Efficient Zeroth-Order LLM Fine-Tuning via Large Learning Rates
- Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling
- TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge
Source: arXiv cs.LG | 2026-10-14