Is the Modality Gap a Bug or a Feature? A Robustness Perspective

arXiv:2603.29080v2 Announce Type: replace Abstract: Many modern multi-modal models (e.g. CLIP) seek an embedding space in which the two modalities are aligned. Somewhat surprisingly, almost all existing models show a strong modality gap: the distribution of images is well-separated from the distribution of texts in the shared embedding space. Despite a series of recent papers on this topic, it is still not clear why this gap exists nor whether closing the gap in post-processing will lead to better performance on downstream tasks. In this paper we show that under certain conditions, minimizing the contrastive loss yields a representation in which the two modalities are separated by a global gap vector that is orthogonal to their embeddings. We also show that under these conditions the modality gap is monotonically related to robustness: decreasing the gap does not change the clean accuracy of the models but makes it less likely that a model will change its output when the embeddings are perturbed. Our experiments show that for many real-world VLMs we can significantly increase robustness by a simple post-processing step that moves one modality towards the mean of the other modality, without any loss of clean accuracy.

Related

• Can Textual Reasoning Improve the Performance of MLLMs on Fine-grained Visual Classification?
• Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM
• CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs
• Robustness Evaluation of a Foundation Segmentation Model Under Simulated Domain Shifts in Abdominal CT: Implications for Health Digital Twin Deployment
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models

Source: arXiv cs.CV | 2026-04-29