BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

arXiv:2603.18019v2 Announce Type: replace Abstract: Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.

Related

• SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
• PeReGrINE: Evaluating Personalized Review Fidelity with User Item Graph Context
• Sell More, Play Less: Benchmarking LLM Realistic Selling Skill
• Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Study
• FinTruthQA: A Benchmark for AI-Driven Financial Disclosure Quality Assessment in Investor -- Firm Interactions
• E2Edev: Benchmarking Large Language Models in End-to-End Software Development Task

Source: arXiv cs.CL | 2026-04-10