VIP: Visual-guided Prompt Evolution for Efficient Dense Vision-Language Inference

arXiv:2605.12325v1 Announce Type: new Abstract: Pursuing training-free open-vocabulary semantic segmentation in an efficient and generalizable manner remains challenging due to the deep-seated spatial bias in CLIP. To overcome the limitations of existing solutions, this work moves beyond the CLIP-based paradigm and harnesses the recent spatially-aware dino.txt framework to facilitate more efficient and high-quality dense prediction. While dino.txt exhibits robust spatial awareness, we find that the semantic ambiguity of text queries gives rise to severe mismatch within its dense cross-modal interactions. To address this, we introduce extcolor{oursblue}{extbf{VI}}sual-guided extcolor{oursblue}{extbf{P}}rompt evolution (extcolor{oursblue}{extbf{extit{VIP}}}) to rectify the semantic expressiveness of text queries in dino.txt, unleashing its potential for fine-grained object perception. Towards this end, VIP integrates alias expansion with a visual-guided distillation mechanism to mine valuable semantic cues, which are robustly aggregated in a saliency-aware manner to yield a high-fidelity prediction. Extensive evaluations demonstrate that VIP: ing{182} surpasses the top-leading methods by 1.4% sim 8.4% average mIoU, ing{183} generalizes well to diverse challenging domains, and ing{184} requires marginal inference time and memory overhead. href{https://github.com/MiSsU-HH/VIP}{Our code is publicly available at GitHub faGithub}.

Source: arXiv cs.CV | 2026-05-13