Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging

arXiv:2609.18688v1 Announce Type: cross Abstract: AI models for multimodal medical imaging must balance modality-specific specialization with cross-modal shared representations, a trade-off that pure Mixture-of-Experts (MoE) architectures currently fail to satisfy. Expert-based routing improves in-domain learning but may sacrifice cross-modal signals, which appear particularly important for rare (low-prevalence) pathologies in our experiments. To resolve this, we introduce Generalist-Specialist-MoE (GS-MoE), a two-branch (MoE) architecture that couples a cross-modal generalist model with distinct modality-specific specialists (experts) via domain-constrained feature fusion. On RadImageNet (1.35M images, 165 pathologies, three modalities), GS-MoE recovers detection of six low-prevalence pathologies on which every baseline scores F1 = 0, with per-class gains up to +0.60 F1. It attains this while even slightly exceeding dense and specialist-only MoE aggregate baselines (MCC 0.770), while using {sim}53% fewer active parameters at inference than the strongest investigated dense model.

Related

• MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
• LongMoE: Longitudinal Multimodal Learning via Trajectory-Aware Mixture-of-Experts
• ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
• SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Experts

Source: arXiv cs.AI | 2026-09-17