Similar Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents

arXiv:2607.10198v2 Announce Type: replace Abstract: Search APIs expose ranked snippets, URLs, and metadata on which agents decide whether to answer, search again, or fetch pages. We evaluate these interfaces as decision surfaces on a fixed sample of 100 questions from the 254-question SealQA-Hard subset, using one frozen GPT-5.4 agent, a fixed orchestration harness, and a shared page-fetch backend across Brave, Tavily, and Firecrawl. A Kimi-K2.6 oracle labels visible URL-level evidence; a separate answer audit yields 25, 25, and 26 correct answers out of 100. These counts indicate similar observed accuracy but do not establish equivalence. Under the tested configurations, Brave exposes more pre-fetch support alongside a larger snippet surface; Tavily has a larger rank-1 share among trajectory-pooled supporting observations; and Firecrawl is associated with broader exploration. First-search query-level metrics distinguish support availability from ranking, while contradiction exposure complements contradiction-to-gold ratios. A retrospective oracle union covers 44/100 questions, versus 26/100 for the best individual provider: 18 percentage points of headroom. The observed evidence and action differences motivate evaluating search APIs jointly with agent policy and retrieval budget.

Related

• A Four-Condition Diagnostic Protocol for Evidence Utilization in Long-Context and Retrieval-Augmented Language Models
• DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
• Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M+ Real Search Requests
• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards
• TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents

Source: arXiv cs.CL | 2026-09-15