Model Releases
If your benchmark relies on a static dataset or sampling from a static distribution densely known at training time, then it is fundamentally…
If your benchmark relies on a static dataset or sampling from a static distribution densely known at training time, then it is fundamentally measuring memorization/retrieval. Which might be fine if yo
If your benchmark relies on a static dataset or sampling from a static distribution densely known at training time, then it is fundamentally measuring memorization/retrieval. Which might be fine if you're looking for a retrieval benchmark! But don't confuse it with intelligence.
Source: Francois Chollet (X) | 2026-06-26