Safety
Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment
arXiv:2607.16841v2 Announce Type: replace Abstract: Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding. However, they remain prone to hallucinat
arXiv:2607.16841v2 Announce Type: replace Abstract: Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding. However, they remain prone to hallucinations, generating responses that are inconsistent with the visual evidence. Existing mitigation methods largely address language-prior bias or cross-modal imbalance, while progressive visual degradation across perception and memory remains underexplored. In this work, we propose Saliency-Driven Perceptual Realignment (SDPR), a training-free framework that mitigates the degradation of visual awareness throughout inference. Specifically, we first introduce saliency-driven attention redistribution to release attention hijacked by non-semantic sink tokens, thereby recovering critical visual evidence. Second, we identify spatial distortion in the KV cache and propose saliency-driven cache alignment to preserve query-relevant visual features during generation. Finally, we introduce prior-constrained contrastive decoding to penalize unfaithful predictions induced by dominant language priors. Our proposed SDPR is robust against hallucinations due to its holistic alignment of visual awareness across the entire generative trajectory. Extensive experiments across diverse LVLM architectures show that SDPR outperforms state-of-the-art methods on both hallucination and general-purpose benchmarks, requiring no additional training and incurring minimal runtime overhead. The code is available href{https://github.com/PengSyuChen/SDPR}{olor{blue}{here}}.
Related
- Mitigating Visual Hallucinations in Multimodal Systems through Retrieval-Augmented Reliability-Aware Inference
- VADER: Adaptive Debiasing for Hallucination Mitigation in Video Large Language Models
- Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings
- Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs
- Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs
Source: arXiv cs.CV | 2026-08-20