Really strong benchmark paper on coding agents. Claude Opus 5 running under Claude Code passes 23.9% of the evaluations. An expert human ref…

Really strong benchmark paper on coding agents. Claude Opus 5 running under Claude Code passes 23.9% of the evaluations. An expert human reference scores 82.2%. Here is what the task actually is. The developer agent is dropped into a client engagement. It gets the records a business really keeps, a client who holds the requirements and will answer questions, a production API that operations has to run through, an inherited codebase, and hard limits on serving cost and model choice. From that it has to deliver a working customer service agent. The score comes from deploying the agent it built against held-out simulated users. 53 tasks across four domains. The failures look familiar to anyone who has done this work. The models query the business records shallowly instead of reading them properly. They tell the client almost nothing. They experiment very little with agent architecture or serving spend, and ship the first design that runs. Existing coding benchmarks score a patch. This one scores whether an agent can do the cooperative part of the job. Paper: https://academy.dair.ai/papers/bench-an-environment-for-end-to-end-realistic-agent-construction-2609.04611

Source: DAIR.AI (X) | 2026-09-07

Related