Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework

arXiv:2602.19549v2 Announce Type: replace Abstract: Visual Document Retrieval (VDR), which aims to retrieve relevant pages within vast corpora of visually-rich documents, is of significance in current multimodal retrieval applications. The state-of-the-art multi-vector paradigm excels in performance but suffers from prohibitive overhead, a problem that current efficiency methods like pruning and merging address imperfectly, creating a difficult trade-off between compression rate and feature fidelity. To overcome this dilemma, we introduce Prune-then-Merge, a novel two-stage framework that synergizes these complementary approaches. Our method first employs an adaptive pruning stage to filter out low-information patches, creating a refined, high-signal set of embeddings. Subsequently, a hierarchical merging stage compresses this pre-filtered set, effectively summarizing semantic content without the noise-induced feature dilution seen in single-stage methods. Extensive experiments on 29 VDR datasets demonstrate that our framework consistently outperforms existing methods, significantly extending the near-lossless compression range and providing robust performance at high compression ratios.

Related

• Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval
• CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding
• CoRoVA: Compressed Representations for Vector-Augmented Code Completion
• Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies

Source: arXiv cs.CL | 2026-04-21