Lost in Aggregation: How Benchmarks Overlook Irreplaceable Model Strengths

arXiv:2608.18919v1 Announce Type: new Abstract: Tabular machine learning benchmarks typically summarize performance by averaging scores, ranks, or pairwise wins across datasets. Such aggregates are useful for selecting robust default models, but they can obscure a different question: which models are necessary to attain peak performance on particular datasets? We argue that benchmark evaluation should also consider the data-centric peak performance frontier, defined by the best statistically supported performance achieved on each dataset. From this perspective, a model may be irreplaceable, sufficient, redundant, or fallible depending on where it lies on the frontier relative to other models. Applying this framework to the TabArena benchmark, we find that common aggregation metrics are highly correlated and largely measure consistency and avoiding failures, while being much less aligned with dataset-level irreplaceability. Consequently, models performing decently across datasets without ever being the best choice are rewarded while models with unique dataset-specific strengths appear mediocre under aggregation. Hence, benchmark progress should be measured not only by improvements on aggregation metrics but also by whether new models expand the set of attainable peak performances across datasets.

Related

• Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings
• Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study
• TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks
• Enhancing Automated Machine Learning via Homogeneous Train-Test Splitting Methods

Source: arXiv cs.LG | 2026-08-20