Expert-Guided Multimodal Fusion for Unified Emotion and Sentiment Analysis

arXiv:2601.07565v2 Announce Type: replace-cross Abstract: Multimodal emotion understanding requires the integration of heterogeneous data sources, including text, audio, and visual modalities, while simultaneously addressing discrete emotion recognition and continuous sentiment analysis. We propose EGMF, a unified framework that combines expert-guided multimodal fusion with large language models to achieve superior performance across both tasks. At the core of our framework is a multi-scale expert network, comprising a local expert for capturing subtle emotional nuances, a semantic correlation expert for modeling cross-modal relationships, and a global context expert for understanding long-range dependencies. These experts are adaptively integrated via hierarchical dynamic gating, enabling context-aware feature selection and modality weighting. The enhanced multimodal representations are seamlessly incorporated into the language model through pseudo token injection and prompt-based conditioning, allowing a single generative framework to handle both classification and regression tasks. We employ parameter-efficient LoRA fine-tuning to maintain computational efficiency. Extensive experiments on bilingual benchmark datasets (MELD, CHERMA, MOSEI, SIMS-V2) demonstrate that EGMF outperforms state-of-the-art methods in terms of accuracy, cross-lingual robustness, and the discovery of universal patterns in multimodal emotional expressions.

Source: arXiv cs.AI | 2026-08-11