RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models

arXiv:2609.02731v1 Announce Type: cross Abstract: Large vision-language models have achieved remarkable success in vision-language tasks. However, they remain prone to Visual Hallucinations (VHs), undermining their reliability in real-world applications. Existing solutions typically require curated datasets, additional training, or multi-round decoding, resulting in considerable computational overhead. In this paper, we propose extbf{RVSD} (nderline{R}etrieval nderline{V}ision nderline{S}parse nderline{D}ecoding), a training-free and plug-and-play decoding framework that, for the first time, unifies token sparsification and extbf{Semantic-Space Visual Retrieval} (SSVR) within a single decoding pass. Within RVSD, we introduce a extbf{semantics-directed token selection} strategy that selectively sparsifies redundant tokens while preserving critical visual information. We further propose the SSVR mechanism, which reformulates visual compensation as an on-demand cross-modal retrieval process within a shared semantic space. Extensive experiments demonstrate that RVSD achieves state-of-the-art performance in mitigating VHs while maintaining robust suppression capabilities under long-context generation settings. Our code is available here.footnote{https://github.com/canjie-liu/RVSD}

Related

• SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
• SSSD: Simply-Scalable Speculative Decoding
• ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
• PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding

Source: arXiv cs.AI | 2026-09-03