From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation

arXiv:2608.26856v1 Announce Type: cross Abstract: Although Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in Medical Visual Question Answering (Med-VQA), their reliance on global image features often lacks precise pixel-level grounding, thereby limiting clinical trustworthiness. To bridge the semantic gap between high-level clinical reasoning and spatial localization, we propose extsc{extsc{MedREAL}} (extbf{Med}ical extbf{RE}asoning-driven extbf{A}nswering and extbf{L}ocalization), a unified framework that seamlessly aligns linguistic reasoning with spatial grounding. Specifically, extsc{MedREAL} introduces extbf{S}eg extbf{A}nchored extbf{R}easoning extbf{P}ooling (SARP) to distill task-relevant semantic evidence directly from exttt{[SEG]} tokens within the MLLM's hidden states. Furthermore, a extbf{R}easoning-to-extbf{V}isual (R2V) fusion mechanism is proposed to effectively inject these reasoning-aware features into a segmentation pipeline for accurate mask decoding. To facilitate this paradigm, we construct MedRAVS-13K, a comprehensive dataset comprising 13,824 expertly validated samples across four diverse imaging modalities. Extensive experiments demonstrate that extsc{MedREAL} significantly outperforms state-of-the-arts, achieving 68.49% gIoU and 70.47% cIoU on benchmark evaluations. By generating evidence masks that are strictly consistent with textual diagnoses, extsc{MedREAL} provides a robust, interpretable framework for reasoning-driven medical image analysis.

Related

• A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering
• LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
• Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering
• Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering

Source: arXiv cs.AI | 2026-08-28