The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models

arXiv:2604.14363v1 Announce Type: new Abstract: Multimodal language models systematically underperform on visual perception tasks, yet the structure underlying this failure remains poorly understood. We propose centroid replacement, collapsing each token to its nearest K-means centroid, as a controlled probe for modal dependence. Across seven models spanning three architecture families, erasing text centroid structure costs 4imes more accuracy than erasing visual centroid structure, exposing a universal imbalance where language representations overshadow vision even on tasks that demand visual reasoning. We exploit this asymmetry through text centroid contrastive decoding, recovering up to +16.9% accuracy on individual tasks by contrastively decoding against a text-centroid-erased reference. This intervention varies meaningfully with training approaches: standard fine-tuned models show larger gains (+5.6% on average) than preference-optimized models (+1.5% on average). Our findings suggest that modal competition is structurally localized, correctable at inference time without retraining, and quantifiable as a diagnostic signal to guide future multimodal training.

Related

• Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs
• Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models
• UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding
• TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning
• ActFER: Agentic Facial Expression Recognition via Active Tool-Augmented Visual Reasoning

Source: arXiv cs.CL | 2026-04-17