Logit Lens Supervision for Patch-Level Explanations in Vision-Language Models

arXiv:2602.01530v2 Announce Type: replace Abstract: Modern autoregressive Vision-Language Models (VLMs) can generate fluent answers while their visual-token representations become weakly tied to the image regions from which they originate. This limits patch-level explainability: a visual token should remain interpretable as the image patch it represents. We study this issue through Logit Lens maps, obtained by projecting each visual-token embedding through the LLM vocabulary head to measure how strongly each image patch is associated with a queried textual concept, such as "cat". We introduce Logit Lens Loss (LLL), a lightweight auxiliary objective that preserves localized visual semantics by directly aligning object-related visual tokens with the vocabulary concepts describing their image regions. LLL requires no architectural modification, mask decoder, or large-scale retraining. We evaluate LLL through both explanation quality and downstream performance. Across LLaVA-v1.5-7B and Qwen2.5-VL-7B, LLL yields sharper object confidence maps, improves grounding and hallucination metrics, transfers to zero-shot pointing, and preserves VQA performance.

Related

• HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Models
• Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs
• Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression
• Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Models

Source: arXiv cs.CV | 2026-08-12