Agents
We let open models access the outside world during evals on tasks from public benchmarks. Here's what happened: Most went and found upstream…
We let open models access the outside world during evals on tasks from public benchmarks. Here's what happened: Most went and found upstream commits containing fixes for the tasks they were supposed t
We let open models access the outside world during evals on tasks from public benchmarks. Here's what happened: Most went and found upstream commits containing fixes for the tasks they were supposed to solve. And across every model, scores were higher when the agent found the fix. For some, that gap is huge: • GPT 5.6 Luna: 0.40 → 0.72 • GLM-5.3: 0.60 → 0.84 • MiniMax M3: 0.31 → 0.63 Introduces big questions about what evals are measuring: ability to code, or ability to find an answer? Evaluated on tasks from SWE-rebench, SWE-smith, Atlas QnA and more.
Related
- Curious finding while creating evals and benchmarks for long-horizon (100+ turn) agents While it’s generally thought that a direct swap to o…
- 1/5 We hit SOTA performance on BrowseComp-Plus with 95.18% accuracy using AI21 Maestro’s agent optimization. Here’s how we automated the sea…
- LiveEvalBench: Toward Open-World Evaluation for Web Generation
Source: AI21 Labs (X) | 2026-09-29