UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding

arXiv:2604.14113v1 Announce Type: cross Abstract: GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layouts. Test-time zoom-in methods improve localization by cropping and re-running inference at higher resolution, but apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case. We propose extbf{UI-Zoomer}, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate fuses spatial consensus among stochastic candidates with token-level generation confidence to selectively trigger zoom-in only when localization is uncertain. When triggered, an uncertainty-driven crop sizing module decomposes prediction variance into inter-sample positional spread and intra-sample box extent, deriving a per-instance crop radius via the law of total variance. Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements over strong baselines across multiple model architectures, achieving gains of up to +13.4%, +10.3%, and +4.2% respectively, with no additional training required.

Related

• Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models
• Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding
• A Domain-Specific Language for LLM-Driven Trigger Generation in Multimodal Data Collection
• Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs

Source: arXiv cs.CL | 2026-04-16