Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs

arXiv:2601.13707v2 Announce Type: replace Abstract: Hallucinations in large vision--language models (LVLMs) often arise when language priors dominate over visual evidence, leading to object misidentification and visually inconsistent descriptions. We address this problem by framing hallucination mitigation as contrastive guidance that steers generation toward visually grounded and semantically faithful text. We propose Attention-space Contrastive Guidance (ACG), a training-free, single-pass method that operates directly in self-attention layers, where hallucination-inducing cross-modal biases emerge. ACG constructs both image-conditioned and approximate text-only attention paths within a single forward pass, enabling efficient guidance before errors accumulate at the output layer. Because this masking-based surrogate can introduce approximation bias, we further apply a lightweight orthogonal projection that suppresses components aligned with the text-only path, yielding a more visually grounded correction. Experiments on CHAIR and POPE show that ACG improves faithfulness over existing training-free baselines while maintaining caption quality, reducing latency by up to 2imes compared to multi-pass contrastive decoding methods.

Related

• See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment
• Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
• Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Models
• To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs

Source: arXiv cs.CV | 2026-04-21