Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

arXiv:2610.02123v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) architectures scale model capacity through sparse computation, routing each token through only a small subset of experts. In this work, we explore whether this sparsity gives rise to emergent intrinsic organization in multimodal MoEs. We find that experts develop strong semantic specialization across modalities and domains despite not being explicitly trained for modularity. Building on this structure, we introduce ExpertLens, a data-free method that identifies domain-specialized experts directly from pretrained model weights by decoding router weights into semantically meaningful vocabulary tokens. We leverage this specialization for efficient multimodal adaptation by selectively fine-tuning experts relevant to a target domain. Across math, medical, and remote sensing tasks, ExpertLens matches or surpasses full fine-tuning while updating only 21.7 - 47.0% of model parameters and achieving a 4.0x average training speedup, and outperforms LoRA in both adaptation performance and training efficiency. These results show that sparsity introduced for efficiency can give rise to semantic modularity that is directly useful for efficient adaptation.

Related

• Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models
• DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
• TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation
• Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference

Source: arXiv cs.CV | 2026-10-02