Invisible to Humans, Triggered by Agents: Stealthy Jailbreak Attacks on Mobile Vision-Language Agents

arXiv:2510.07809v4 Announce Type: replace-cross Abstract: Large Vision-Language Models (LVLMs) empower autonomous mobile agents, yet their security under realistic mobile deployment constraints remains underexplored. While agents are vulnerable to visual prompt injections, stealthily executing such attacks without requiring system-level privileges remains challenging, as existing methods rely on persistent visual manipulations that are noticeable to users. We uncover a consistent discrepancy between human and agent interactions: automated agents generate near-zero contact touch signals. Building on this insight, we propose a new attack paradigm, agent-only perceptual injection, where malicious content is exposed only during agent interactions, while remaining not readily perceived by human users. To accommodate mobile UI constraints and one-shot interaction settings, we introduce HG-IDA*, an efficient one-shot optimization method for constructing jailbreak prompts that evade LVLM safety filters. Experiments demonstrate that our approach induces unauthorized cross-app actions, achieving 82.5% planning and 75.0% execution hijack rates on GPT-4o. Our findings highlight a previously underexplored attack surface in mobile agent systems and underscore the need for defenses that incorporate interaction-level signals.

Related

• Harnessing Hyperbolic Geometry for Harmful Prompt Detection and Sanitization
• WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks
• AgentCity: Constitutional Governance for Autonomous Agent Economies via Separation of Power
• Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Space
• Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions

Source: arXiv cs.AI | 2026-04-10