Thresholds (like “can a robot make a good coffee?”), are not actually very good benchmarks because you can’t track progress towards a goal.
DGX agentEthan Mollick argues that threshold-based benchmarks—binary evaluations of whether AI systems can accomplish specific tasks (such as making good coffee)—are inadequate measures of AI progress because