Safety
Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
arXiv:2604.17217v1 Announce Type: new Abstract: Vision-Language Models (VLMs) achieve strong cross-modal performance, yet recent evidence suggests they over-rely on textual descriptions while under-ut
arXiv:2604.17217v1 Announce Type: new Abstract: Vision-Language Models (VLMs) achieve strong cross-modal performance, yet recent evidence suggests they over-rely on textual descriptions while under-utilizing visual evidence -- a phenomenon termed ``text shortcut learning.'' We propose an adversarial evaluation framework that quantifies this cross-modal dependency by measuring accuracy degradation (Drop) when semantically conflicting text is paired with unchanged images. Four adversarial strategies -- shape_swap, color_swap, position_swap, and random_text -- are applied to a controlled geometric-shapes dataset (n{=}1{,}000). We compare three configurations: Baseline CLIP (ViT-B/32), LoRA fine-tuning, and LoRA Optimized (integrating Hard Negative Mining, Label Smoothing, layer-wise learning rates, Cosine Restarts, curriculum learning, and data augmentation). The optimized model reduces average Drop from 27.5% to 9.8% (64.4% relative improvement, p{<}0.001) while maintaining 97% normal accuracy. Attention visualization and embedding-space analysis confirm that the optimized model attends more to visual features and achieves tighter cross-modal alignment.
Related
- Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs
- See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment
- Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
- To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs
Source: arXiv cs.CV | 2026-04-21