RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

arXiv:2610.06243v1 Announce Type: new Abstract: Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoSA (Rotational Sparse Adaptation), which narrows adaptation to a subset of layers at a time. RoSA freezes lower layers close to the input throughout training and rotates a trainable block over later layers, progressively increasing the number of frozen layers close to the input. This design reduces optimizer-state memory, shortens backpropagation, and even forward propagation if activations at the last frozen layer are cached. Because RoSA is orthogonal to the choice of trainable parameterization, it can be combined with PEFT methods or sparse optimizers within each active block. Experiments across multiple LLM architectures and tasks show that RoSA reduces peak memory while maintaining strong fine-tuning performance.

MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation
Continual Fine-Tuning of Large Language Models via Program Memory
Optimizing the Phi-2 Small Language Model for Real-time Chatbot Applications Using Parameter-Efficient Fine-Tuning (PEFT) with QLoRA Quantization
HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation

Source: arXiv cs.LG | 2026-10-06

Related