Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective

arXiv:2505.20816v2 Announce Type: replace Abstract: Recent advances in multimodal question answering have primarily focused on combining heterogeneous modalities or fine-tuning multimodal large language models. While these approaches have shown strong performance, they often rely on a single, generalized reasoning strategy, overlooking the unique characteristics of each modality ultimately limiting both accuracy and interpretability. To address these limitations, we propose MAMMQA, a multi-agent QA framework for multimodal inputs spanning text, tables, and images. Our system includes two Visual Language Model (VLM) agents and one text-based Large Language Model (LLM) agent. The first VLM decomposes the user query into sub-questions and sequentially retrieves partial answers from each modality. The second VLM synthesizes and refines these results through cross-modal reasoning. Finally, the LLM integrates the insights into a cohesive answer. This modular design enhances interpretability by making the reasoning process transparent and allows each agent to operate within its domain of expertise. Experiments on diverse multimodal QA benchmarks demonstrate that our cooperative, multi-agent framework consistently outperforms existing baselines in both accuracy and robustness.

Related

• MATA: Multi-Agent Framework for Reliable and Flexible Table Question Answering
• MASS-RAG: Multi-Agent Synthesis Retrieval-Augmented Generation
• Hear Both Sides: Efficient Multi-Agent Debate via Diversity-Aware Message Retention
• Towards Self-Improving Error Diagnosis in Multi-Agent Systems
• TopoDIM: One-shot Topology Generation of Diverse Interaction Modes for Multi-Agent Systems

Source: arXiv cs.CL | 2026-04-22