alphaTransfer: Coefficient Transfer for Efficient Model Merging

arXiv:2610.07819v1 Announce Type: cross Abstract: Model merging offers a promising solution for combining multiple fine-tuned checkpoints into a single model through parameter arithmetic. However, finding optimal merging coefficients requires an extensive search that becomes prohibitively expensive as models scale in both size and number, due to high memory requirements and combinatorial growth in the search space. We show that, within the same model family, models exhibit highly congruent performance distributions over merging coefficients across different model sizes. This distributional similarity enables a practical paradigm we call extit{alphaTransfer}: searching for optimal coefficients on a small proxy model, then directly transfer them to larger target models. We verify alphaTransfer across multiple merging methods, model families, and tasks. Experimental results demonstrate a 6imes speedup and 70% memory reduction on vision transformers, and a 20imes speedup and 85% memory reduction on large language models, while maintaining comparable performance. Our findings establish alphaTransfer as an efficient and generalizable approach to scaling model merging.

Related

• CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models
• LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging
• DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging
• Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space
• TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging

Source: arXiv cs.CL | 2026-10-07