Model Releases
Fine-Tuning of Transformer models with Frames
arXiv:2608.26430v1 Announce Type: new Abstract: Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained mod
arXiv:2608.26430v1 Announce Type: new Abstract: Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, O(dr), where d is the model's hidden dimension and r is the rank. Our proposal, FrameFT, models the parameter update Delta W with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
Related
- GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioning
- RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models
- Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs
- LoCO: Low-rank Compositional Rotation Fine-tuning
Source: arXiv cs.AI | 2026-08-28