MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search

arXiv:2610.17792v1 Announce Type: new Abstract: GUI grounding enables web agents to translate natural-language instructions into actionable interface locations. However, dense layouts and small, visually similar controls can lead models toward plausible but incorrect elements, while refinement around a single prediction may leave alternative targets unexplored. We propose the Multi-Hypothesis Verification Framework (MHVF) for training-free GUI grounding. It combines multi-hypothesis exploration with local visual verification using a single pretrained vision-language model. To explore alternative targets, MHVF constructs spatial hypotheses from multiple full-screen predictions and examines the corresponding regions through zoomed re-localization. To resolve ambiguity among the resulting candidates, it verifies their agreement with the instruction in marked local views. Verification scores then guide a second round of local search and rank candidates retained from both search levels for final selection. Experiments with five general-purpose and GUI-specific models across eight benchmark settings demonstrate improvements in grounding accuracy in most settings. MHVF improves UI-Venus-Ground-7B by 4.3 and 2.1 percentage points on Mind2Web and VisualWebBench, respectively, and raises Qwen2.5-VL-7B's accuracy on ScreenSpot-Pro from 23.6% to 53.6%.

Related

• See, Point, Refine: Multi-Turn Approach to GUI Grounding with Visual Feedback
• How Auxiliary Reasoning Unleashes GUI Grounding in VLMs
• Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning
• GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction

Source: arXiv cs.CV | 2026-10-14