Rethinking Expert Training for Model Merging with Prompt Learning

arXiv:2607.24465v1 Announce Type: new Abstract: Model merging aims to combine multiple domain-specialized experts trained from a shared foundation model into a single multi-task model. Existing approaches largely focus on improving the merging procedure itself and typically assume experts obtained through full-parameter fine-tuning. In this work, we revisit expert training for model merging. We first show that prompt-based adaptation provides a strong baseline: independently learned prompts can be exploited across tasks while keeping the backbone fixed, avoiding the interference introduced by weight merging. Building on this observation, we introduce Dual-Tuned Experts (DTEs), a two-stage training strategy that first learns prompts and then fine-tunes the vision encoder. This reduces the magnitude of task-specific parameter updates and produces experts with higher merge compatibility. Experiments across multiple CLIP architectures, full fine-tuning, and LoRA experts show that DTEs consistently improve merged performance of standard merging approaches and remain effective even when combining heterogeneous sets of experts.

Related

• Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs
• Closed-Form Spectral Regularization for Multi-Task Model Merging
• Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space

Source: arXiv cs.CV | 2026-07-28