Model Releases
From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature
arXiv:2511.22232v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are increasingly capable in medical imaging, yet most focus on single-image settings. Clinical interp
arXiv:2511.22232v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are increasingly capable in medical imaging, yet most focus on single-image settings. Clinical interpretation often requires integrating evidence across multiple images, such as different modalities, views, or time points. However, large-scale medical multi-image data and training strategies for such reasoning remain limited. We construct PMC-MI, a large-scale resource comprising 234,956 instruction instances derived from biomedical compound figures, with 10,555 multi-subimage instances structured for reinforcement learning and assessed by medical reviewers. We also introduce PMC-MI-Bench, a manually reviewed benchmark separated at the source-article level. We further propose a three-stage training framework, instantiated as M3LLM, combining supervised fine-tuning on multi-image instructions, selection-aware reinforcement learning for question-conditioned visual-evidence selection, and supervised consolidation on a broader instruction mixture. Evaluating M3LLM on PMC-MI-Bench, two public medical benchmarks, longitudinal chest radiographs, and a retrospective dermatology cohort reveals it achieves the highest performance among representative MLLMs. On PMC-MI-Bench, M3LLM scored 81.6, 84.6, and 85.2 on single-subimage, multi-subimage, and relative-position tasks, respectively, outperforming the strongest baselines (75.6, 79.1, and 77.6). It also set new records on the provenance-screened OmniMedVQA (88.6%) and MMMU-Med (68.0%), alongside top performance in two clinical settings after task-specific adaptation. These findings support using literature-derived multi-image supervision and targeted post-training for medical multi-image MLLMs. Data and code are publicly available.
Related
- Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis
- Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI
- From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation
- JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models
Source: arXiv cs.AI | 2026-10-01