Benchmarking AI systems is a continual process that co-evolves with the models. New benchmarks challenge AI capabilities with emerging quest…

Benchmarking AI systems is a continual process that co-evolves with the models. New benchmarks challenge AI capabilities with emerging questions to shape the directions and feedback signal of the research process. Then they adapt as models progress, targeting the residual between AI and human intelligence. We are still working on ARC-AGI-4, which we started developing after releasing ARC-AGI-3 earlier this year. It is coming Q1 2027. We think it's going to be really special.

Related

• Really interesting concept -- instead of optimizing against static benchmarks or internally generated reward models, agents are evaluated th…
• There are only two honest metrics when it comes to benchmarking intelligence: novelty and efficiency. You don't need intelligence to solve a…
• This is the sort of early prediction you can make when you pay close attention to ARC-AGI scores
• Thanks. The fact that it works so well for ARC is also a clue to the extent of generalization in these models. It was not at all obvious tha…

Source: Francois Chollet (X) | 2026-09-03