Applications
The state of public AI benchmarking is dire and is undermining our ability to understand how good AI is now. Most famous measures are maxed …
The state of public AI benchmarking is dire and is undermining our ability to understand how good AI is now. Most famous measures are maxed out, and, as this paper shows, the non-saturated benchmarks
The state of public AI benchmarking is dire and is undermining our ability to understand how good AI is now. Most famous measures are maxed out, and, as this paper shows, the non-saturated benchmarks are riddled with so many errors that they vastly underestimate AI abilities.
Related
- Thresholds (like “can a robot make a good coffee?”), are not actually very good benchmarks because you can’t track progress towards a goal.
- Is the @METR_Evals long horizons measure effectively saturated? No updates to the most famous graph of AI progress since May, but other publ…
- As the benchmarks that test frontier AI on get more complex, we are losing one of the most important aspects of benchmarking: comparisons to…
- As much as the state of benchmarks in AI is flawed, it is so much easier to track AI progress than robotics. Not sure what you can make of a…
Source: Ethan Mollick (X) | 2026-09-16