How Reusable Are Benchmarks with Richer Feedback?

arXiv:2609.32109v1 Announce Type: new Abstract: We study whether benchmarks reliably guide model selection as developers adapt to evaluation feedback across multiple criteria. We find that the worst-case test-set size needed to estimate the best score among k adaptively chosen models, under any convex combination of the criteria, grows exponentially with the number of criteria, reaching the Theta(sqrt{k}) cost of answering k adaptive statistical queries with only O(log k) criteria, at fixed accuracy and confidence. In attacks on multi-task large language model benchmarks with five to ten criteria, feedback restricted to nondominated task profiles produces large reused-to-held-out score gaps and frequent false winners. These results challenge a prominent explanation for prior observed reliable benchmark reuse---that developers mainly respond to convincing improvements over the current best---in rich-feedback settings, while leaving open how often ordinary model development encounters this vulnerability.

Source: arXiv cs.LG | 2026-09-29