Evals ~= Environments…they’re one of the best investments a team can make for improving agents Step 0: Turn On Tracing for Agents Step 1: Po…

Evals ~= Environments…they’re one of the best investments a team can make for improving agents Step 0: Turn On Tracing for Agents Step 1: Point compute at Traces to understand agent behavior, segment useful tasks, and isolate error modes Step 2: Turn Trace Data into Evals/Environments to improve your agent. Evals are basically measures of behavior we want our agents to exhibit in Production. Evals are training data. Because we adjust our agent to pass them, every eval is a vector that shifts overall agent behavior. Step 3: Choose your agent improvement approach - Harness Engineering (gradient free) - SFT/RL model in harness - Both The point of curating evals and environments from trace data is that they better reflect the behavior distribution of what your agent will actually do The point of harness engineering and fine-tuning is to fit your agent to the distribution of the tasks it will see and be good at them —> we’re trying to create specialized intelligence for your task using data that reflects your task Existing benchmarks capture some abilities like long horizon reasoning, tool-use, but tailored data to your task will almost always be better. The tradeoff is that it’s more work to source that data, but it’s usually worth it Step 4: Build the fly-wheel Continue gathering Trace data, generating evals/environments, doing human review, and creating better agents over time. It doesn’t have to be continued “training” of the same agent - it could be training other specialized agents and exposing those as sub-agents in your overall system We’re seeing sparks of this semi-automated loop working today, and investing a bunch in making this entire loop as easy as possible for builders 🚀 Recently, PostTrainBench showed how well AI agents can post-train models. Meta Harness showed that the harness itself can improve. What happens if a harness is improvising itself and the improvised harness is post training language models, all in a loop? @Vtrivedy10 on this:

Related

• What does it actually take to make agents better over time? A system that starts with a trace. You capture traces of agent behavior, enrich …
• tldr > evals are the new training data. instead of updating weights, you're updating the agent harness > problem is agents are famous cheate…
• a useful mental model on how teams can think about good data design to improve their models/agents: Evals ~= Training Data ~= Environments -…
• Data Driven Agent Design with Evals & Hill Climbing Algorithms this is a mental model dump i’ve been thinking through + iterating on as we’r…

Source: Harrison Chase (X) | 2026-04-23