Are Benchmarks Reliable? Toward Structural Diagnosis via Item-Level Capability Boundaries

arXiv:2609.33196v2 Announce Type: replace Abstract: Evaluating large language models (LLMs) relies heavily on benchmark scores, yet aggregate metrics can obscure whether individual items serve their intended measurement purpose. BSDProbe provides an item-level framework for benchmark structural diagnosis that jointly characterizes correctness transitions and their supporting response evidence. Repeated responses along an empirically ordered model axis yield complementary correctness and normalized entropy trajectories, from which four-dimensional profiles of boundary position, boundary width, boundary-signal validity, and order consistency are constructed for benchmark-level aggregation. Experiments on six benchmarks identify varying proportions of potentially anomalous items. With Qwen3 as both the selection and reference axis, BSDProbe achieves a 17.9% mean relative improvement in normalized discrimination ratio over the best baseline per benchmark and matches or exceeds all baselines in ranking agreement with the full benchmark.

Related

• Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
• Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
• Are Multimodal Benchmarks Really Useful? Item-Level Multimodal Benchmark Diagnosis via Structure-Response Co-Calibration
• Uncovering Competency Gaps in Large Language Models and Their Benchmarks

Source: arXiv cs.AI | 2026-10-15