LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights

arXiv:2602.17510v2 Announce Type: replace-cross Abstract: We introduce LoRA-CRAFT (extbf{C}ross-layer extbf{R}ank extbf{A}daptation via extbf{F}rozen extbf{T}ucker), abbreviated CRAFT throughout, an extremely parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors. Existing tensor-based PEFT methods decompose extit{gradient updates}: LoTR applies Tucker decomposition with shared factor matrices, while SuperLoRA groups and reshapes Delta W across layers before applying Tucker decomposition. Separately, methods such as PiSSA apply SVD to extit{pre-trained weights} but operate independently per layer. CRAFT bridges these two lines of work: it performs full Tucker decomposition via Higher-Order SVD (HOSVD) directly on extit{pre-trained weights} organized as cross-layer 3D tensors, freezes all resulting factors, and adapts the model through lightweight trainable transformations applied to each factor matrix. Experiments on the GLUE benchmark using RoBERTa-base and RoBERTa-large, as well as commonsense reasoning benchmarks using LLaMA2-7B and LLaMA3-8B, demonstrate that CRAFT achieves competitive performance with existing methods while requiring only rev{extbf{extremely low Tucker adaptation parameters}}. fixw{On LLaMA3-8B, CRAFT} rev{exceeds the average accuracy of LoRA} extbf{using hundreds of times fewer parameters}fixw{; on LLaMA2-7B the same holds at a 0.252M budget}. Our results suggest that CRAFT's efficiency advantage grows with model scale, as the frozen Tucker factors better capture the richer cross-layer structure of larger pre-trained models.

Source: arXiv cs.AI | 2026-08-18