HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment

arXiv:2608.23921v1 Announce Type: new Abstract: Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, existing methods score each visual token by averaging text-to-visual attention uniformly across all heads, which assumes every head matches the query. However, our empirical analysis shows that misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues. To address this, we propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions. Built on PAQ, our pruning proceeds in three stages. Given a target FLOPs budget, we first partition the transformer layers into groups and allocate a visual token budget to each. Within each group, we then aggregate per-head attention maps via PAQ-weighted softmax into a group-level matrix. Finally, we score visual tokens by this matrix's magnitude and retain the allocated budget per group. By weighting heads with PAQ, our method scores tokens by attention signals that more faithfully reflect prompt relevance, rather than diluting them through uniform averaging. Across 18 benchmarks, our method delivers state-of-the-art trade-offs. Specifically, on LLaVA-1.5-7B (9 tasks), retaining only extbf{5.6%} tokens preserves extbf{99.1%} of the original performance, surpassing the strongest baseline AutoPrune by 4.2 points. Code is available in https://github.com/baokou-fw2/HAP.

Related

• ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs
• Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
• ERASE: Eliminating Redundant Visual Tokens via Adaptive Two-Stage Token Pruning
• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
• An AI4AI Framework for Visual Token Pruning

Source: arXiv cs.CV | 2026-08-26