LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization

arXiv:2506.09373v3 Announce Type: replace-cross Abstract: The advent of autonomous agents is transforming interactions with Graphical User Interfaces (GUIs) by employing natural language as a powerful intermediary. Despite the predominance of Supervised Fine-Tuning (SFT) methods in current GUI agents for achieving spatial localization, these methods face substantial challenges due to their limited capacity to accurately perceive positional data. Existing strategies, such as reinforcement learning, often fail to assess positional accuracy effectively, thereby restricting their utility. In response, we introduce Location Preference Optimization (LPO), a novel approach that leverages locational data to optimize interaction preferences. LPO uses information entropy to predict interaction positions by focusing on zones rich in information. Besides, it further introduces a dynamic location reward function based on physical distance, reflecting the varying importance of interaction positions. Supported by Group Relative Preference Optimization (GRPO), LPO facilitates an extensive exploration of GUI environments and significantly enhances interaction precision. Comprehensive experiments demonstrate LPO's superior performance, achieving SOTA results across both offline benchmarks and real-world online evaluations. Our code will be made publicly available soon, at https://github.com/jqtangust/LPO.

Related

• UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
• WebUncertainty: Dual-Level Uncertainty Driven Planning and Reasoning For Autonomous Web Agent
• EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis
• WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents
• Computer Environments Elicit General Agentic Intelligence in LLMs

Source: arXiv cs.AI | 2026-04-22