Are we missing a benchmark for agent runtimes, not just models?

We have SWE-bench, Terminal-Bench, OSWorld, BrowseComp, etc. But I haven’t seen a good apples-to-apples benchmark for platforms like OpenAI Agents, Anthropic’s agent stack, AWS AgentCore, Google’s agent platform, and local-alternatives like LangGraph, etc. What I’d want measured: - task success rate - cost per successful task - wall-clock time - tool calls / retries - reliability over long-running tasks - (optionally) human interventions required The really interesting experiment would control for both sides: - Same model, different harness → how much does the runtime matter? - Same harness, different model → how much does the model matter? Feels like we’re increasingly evaluating “model + harness” systems, but our benchmarks still mostly treat the model as the unit of comparison. Does something like this already exist? submitted by /u/Balance- [link] [comments]

Source: r/LocalLLaMA | 2026-09-11