Lit Review on Benchmarking LLMs Running in your phone!: MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments

Back to reading about LLMs as agents on your phone doing GUI tasks! This time I read about MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments and this paper forms the basis of the benchmark I am currently making because it involves two new innovations: User Interaction Tasks MCP Tasks These two provides new axes of evaluations for such GLLMs as GUI Agents on phones and it separates them from the noise completely because these new axes along imo. So, in total 201 tasks and about 20 apps in general spanning across comms, messaging, productivity and many more with less than ~5% of system apps (basically this means more apps that we humans use on a daily basis -- but those are rather open source version of them which is a downside)! They used a planner-executor arch with the planner being a VLM model (screenshots only being fed -- no a11y tree) that outputs actions in natural text like "Click on the 'Send button' and this gets sent to a grounding model that output its precise coordinates in (x,y) format. So, User Interaction Tasks involve tasks which one key info missing and the planner needs to understand this and ask the user but here the user is a GPT4 model as a replacement for human! Next, MCP tools these are very important since MCP tools allows for complex data gathering, reading and actions that are not possible by tapping etc actions so these provide one shot result for the red tasks thus saving time than doing slow GUI actions! Some of them used are Github, arXiv, etc tools Now, the avg results is about ~52% for the best VLM+Grounding model combo which is Gemini-3-Pro + UI-Inst-7B and the e2e GUI only models perf wayyy worse. The rest of the dataset dist stats and results is attached as images but yes the model mainly failed a lot in the two new axes tasks. submitted by /u/East-Muffin-6472 [link] [comments]

Source: r/LocalLLaMA | 2026-09-07