SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis

arXiv:2601.18305v2 Announce Type: replace Abstract: Despite numerous Graphical User Interface (GUI) agents claiming to automate user interaction tasks, to date, few achieve satisfactory interaction capability with human users in real-world scenarios. Through empirical analysis, this paper identifies the root cause of the limited interaction capability as the rigid swipe execution. In particular, unlike humans, who perform swipes with fine-grained control over trajectory, speed, and timing, existing agents can only conduct simplistic, deterministic swipe behaviors, leading to frequent failures on complicated user-like interaction tasks. Due to the lack of open-source human-like swipe training data, we propose SwipeGen, the first tool for synthesizing diverse and human-like swipe interactions, and SwipeBench, the first benchmark for evaluating agents' swipe interaction quality. Extensive experiments show that SwipeGen can improve the swipe execution success rate of existing agents by up to 2.46x. Our code, dataset, and model are available at https://github.com/TSKGHS17/SwipeGen.

Related

• How Auxiliary Reasoning Unleashes GUI Grounding in VLMs
• See, Point, Refine: Multi-Turn Approach to GUI Grounding with Visual Feedback
• GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
• Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents

Source: arXiv cs.CV | 2026-08-21