Research
From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankers
arXiv:2609.00667v1 Announce Type: cross Abstract: Large vision-language models used as listwise rerankers must jointly process visual tokens from tens of candidates per query, making token pruning ess
arXiv:2609.00667v1 Announce Type: cross Abstract: Large vision-language models used as listwise rerankers must jointly process visual tokens from tens of candidates per query, making token pruning essential for practical deployment. Existing pruning methods retain tokens by attention saliency, yet we show that saliency is systematically misaligned with ranking contribution: visually prominent tokens often capture order-neutral patterns shared across candidates. This mismatch is layer-dependent: saliency becomes informative only where attention is concentrated, and normalized attention entropy diagnoses the reliability shift (Pearson r=0.87). We propose RaDiCal (Rank-Discriminative Calibration), a training-free framework that uses normalized attention entropy to decide when saliency can be trusted, fusing it with an attention-free rank-discriminative prior and selecting pruning layers from the same trust landscape. Across three retrieval benchmarks and multiple VLM architectures, RaDiCal matches Dense MRR@10 on Flickr30K and surpasses it on MSCOCO at a 20% token budget, ranks first among all pruning methods on FashionIQ, and holds within 1.2 pp on Flickr30K and MSCOCO at 10% retention. It cuts FLOPs by 39--45% and delivers 1.28--1.45imes measured speedups across two VLM architectures without dataset-specific retuning.
Related
- Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs
- Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models
- HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Models
Source: arXiv cs.CV | 2026-09-02