Agents
great write up, evals are hard! here are 2 broad buckets we use to evaluate agents: 1. Measure the State of the World 2. Agent as a Judge on…
great write up, evals are hard! here are 2 broad buckets we use to evaluate agents: 1. Measure the State of the World 2. Agent as a Judge on the Trajectory 1. Measure the state of the environment befo
great write up, evals are hard! here are 2 broad buckets we use to evaluate agents: 1. Measure the State of the World 2. Agent as a Judge on the Trajectory 1. Measure the state of the environment before and after the Agent does the Task. @harborframework and containerized Evals make this easier. The agent thinks in operating in a real world and we need to set up that world (files, installed tools, security boundaries, instructions) such that it mirrors what the agent will do in the real world. Every action the agent takes alters the world in some way. And success on a task means running a verification step on the final state of the world after the agent is done to see if this state corresponds to a successful pass. The tricky thing here is calibrating when the agent produces a final state that’s technically correct but your verifier didn’t forsee - you often want to allow this to pass or make sure the correct end state is capturable by your verifier. 2. Judge the trajectory -> Correctness is not the only factor - cost, latency, interpretability matter too. Those require understanding the action space The right answer is a good signal, but if the agent got there by cheating the Task then that means our agent is misaligned AND our task is bad because it allows cheating. If the agent is inefficient with cost/tokens, then that’s a signal that we may need to imbue more priors to help it find the correct trajectory (could be better prompting) we’ll be doing a bunch more educational content and open sourcing our tooling, reach out if there’s anything you want to see in particular! this should be accessible and easy for every team to do! Why is evaluating agents so difficult relative to evaluating a standard LLM? An LLM generates a single response to a prompt. An agent instead interacts with an environment by reasoning, calling tools, observing the results, and repeating. Rather than evaluating a single output, w…
Related
- 1/ you are probably overcomplicating the environment that your agent has to work with when running evals
- Evaluate agents with Harbor + LangChain ‼️
- Great new drop by @LangChain’s @hwchase17 “If you do not know what the agent saw, what it did, and what happened next, you cannot reliably k…
- you can condense long horizon evals with agents into smaller subsets that still let you test intended behavior. i'm currently evaluating an …
- Read more on what we've learned while developing long-horizon agent evals:
Source: Harrison Chase (X) | 2026-08-01