NeuCME: Toward Dynamic Multimodal Continual Learning via Neural Combinatorics of Multiple Experts

arXiv:2609.07009v1 Announce Type: cross Abstract: Multimodal continual learning has recently shown great potential for developing agents with human-like intelligence by continuously learning new tasks across multiple modalities. However, existing methods typically assume that the set of modalities per task is predefined and fixed. In this paper, we investigate a more realistic learning setting, referred to as dynamic multimodal continual learning, in which the set of modalities may vary across tasks rather than remaining fixed. This setting involves two primary challenges: (i) spatio-temporal catastrophic forgetting and (ii) adaptive multimodal fusion. To address these challenges, we propose NeuCME (as shorthand for extbf{Neu}ral extbf{C}ombinatorics of extbf{M}ultiple extbf{E}xperts), a novel framework designed to effectively learn and integrate knowledge across tasks with varying modalities. The proposed NeuCME model comprises three key components, namely modality-combinational rehearsal, multi-gated mixture-of-experts, and task relevance-guided distillation. Furthermore, we formulate an evaluation metric to quantify the dynamism of task sequences and then set up a comprehensive benchmark with different degrees of dynamism. Extensive experiments using four real-world datasets demonstrate that the proposed NeuCME outperforms state-of-the-art methods markedly.

Related

• Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts
• CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detection
• Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models

Source: arXiv cs.CV | 2026-09-09