Rethinking Adapter Placement: A Dominant Adaptation Module Perspective

arXiv:2605.06183v2 Announce Type: replace Abstract: Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method that places trainable low-rank adapters into frozen pre-trained models. Recent studies show that using fewer LoRA adapters may still maintain or even improve performance, but existing methods still distribute adapters broadly, leaving where to place a limited number of adapters to maximize performance largely open. To investigate this, we introduce extbf{PAGE} (extbf{P}rojected extbf{A}dapter extbf{G}radient extbf{E}nergy), a gradient-based sensitivity probe that estimates the initial trainable gradient energy available to each candidate LoRA adapter. Surprisingly, we find that PAGE is highly concentrated on a single shallow FFN down-projection across two model families and four downstream tasks. We term this module the extbf{dominant adaptation module} and show that its layer index is architecture-dependent but task-stable. Motivated by this finding, we propose extbf{DomLoRA}, a placement method that places a single adapter at the dominant adaptation module. With only extbf{0.7%} of vanilla LoRA's trainable parameters, DomLoRA outperforms it on average across downstream tasks, including instruction following, mathematical reasoning, coding, and multi-turn conversation. This method also matches or improves other LoRA variants and reduces training time by up to extbf{2.74}imes compared with broad placement, supporting the dominant adaptation module perspective as a practical placement guideline.

Related

• Selective LoRA for Visual Tokens and Attention Heads
• Semantic Adapter Routing with Fine-Tuning Task Embeddings
• HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
• LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

Source: arXiv cs.AI | 2026-10-07