MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models

arXiv:2608.10635v1 Announce Type: cross Abstract: Medical Vision-Language Models (Med-VLMs) excel at verbalizing visual content, yet precise visual perception, segmentation, and grounding remain challenging. Existing approaches either verbalize regions as coordinate strings or rely on external modules that decouple perception from understanding, creating representation gaps for region-language alignment. We present MedUP, a Med-VLM that natively unifies perception and understanding within a shared token space. At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary, enabling the model to seamlessly interleave mask tokens with text. We curate UniMed-Train, a 1.84M-instance corpus spanning text-guided segmentation, region-grounded understanding, medical VQA and CoT-based segmentation, and introduce UniMed-Bench for unified evaluation. Extensive experiments show that MedUP outperforms native, agentic, and dual-decoder Med-VLMs across all tasks while remaining competitive with specialist segmentors, demonstrating the strong potential of unified understanding and perception modeling.

Related

• DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation
• CURE: Curriculum-guided Multi-task Training for Reliable Anatomy Grounded Report Generation
• Enhancing the Safety of Medical Vision-Language Models by Synthetic Demonstrations
• Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine

Source: arXiv cs.AI | 2026-08-12