Model Releases
MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents
arXiv:2508.13186v2 Announce Type: replace-cross Abstract: AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However,
arXiv:2508.13186v2 Announce Type: replace-cross Abstract: AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However, existing benchmarks such as BrowseComp primarily focus on textual content, overlooking the prevalence of multimodal content. To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising 400 challenging, hand-crafted questions designed to evaluate multimodal retrieval and reasoning capabilities. Unlike prior work, MM-BrowseComp incorporates visual prompts and necessitates the extraction of key evidence from web images and videos to complete questions, rendering text-only approaches insufficient. Additionally, we provide a verified checklist for each question, enabling fine-grained analysis of multimodal dependencies and reasoning paths. Our comprehensive evaluation of 27 state-of-the-art models reveals that even leading models like GPT-5-High with tools achieve only 24.25% accuracy, highlighting the suboptimal multimodal browsing capabilities, establishing MM-BrowseComp as a rigorous new standard for the field.
Related
- WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance
- Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation
- Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History
- Towards Long-horizon Agentic Multimodal Search
Source: arXiv cs.AI | 2026-09-01