Hot take: evals are the #1 thing keeping companies from becoming AI-native. If you want to understand evals, read this post & then the accom…

Hot take: evals are the #1 thing keeping companies from becoming AI-native. If you want to understand evals, read this post & then the accompanying guide. Tweaking prompts and trying a few inputs by hand only gets you so far; evals are what get your agents ready for deployment and keep them working once they're live. First, some definitions... - An eval: a grader applied to a trace. - A grader: anything that scores how well the agent did, from a simple code check to an LLM acting as a judge. - A trace: the full record of one agent run: the input, every step it took, and the output. For example, a customer asks "Where is my refund?" The agent calls search_orders, then issue_refund, and replies "Refund sent, 3–5 days." Next, let's dive into graders... Graders come in a few families, and most real setups run several on the same trace. 1) Deterministic checks. Plain code, no model involved. Did the agent call issue_refund exactly once, with the same order ID it looked up? A double refund or a refund on the wrong order fails instantly. 2) Reference-answer graders. Compare the output to a reference answer stored on the case, by exact match or with an LLM judge checking whether the meaning matches. 3) Rubric graders. An LLM judge scores the output against criteria, with no reference answer needed. Is the reply concise? Does it confirm what happened? 4) Trajectory graders. These grade the steps the agent took, not just its final answer. Did it look up the order before issuing the refund? A run that calls issue_refund without checking the order first fails even if the final reply reads perfectly. What are online vs. offline evals? Online evals watch production. Offline evals run in your lab: on demand or in CI, before anything ships. - You need both. Online without offline means you can see problems but can't fix them safely. - Offline without online means you're testing the scenarios you imagined, not the ones your users actually create. - What connects the two is promotion, turning a bad production trace into an offline case. Next, eval-driven development... Eval-driven development is test-driven development for agent behavior. 1) Define the case and its pass criteria. 2) Run it and watch it fail. 3) Change the agent. 4) Run it again. Repeat steps 3 and 4 until it passes. 5) Run the full dataset, not just this case, to catch regressions. Run each case several times (k) and set the bar explicitly: all-of-k for high-stakes behavior where a single failure is unacceptable, or n-of-k, like 2 of 3, where some variance is acceptable. Put together, online evals find a problem, promotion turns it into a case, the loop fixes it offline, and the growing dataset makes sure it stays fixed. Read our full evals guide here: Hot take: evals are the #1 thing keeping companies from becoming AI-native. If you want to understand evals, read this guide on eval-driven development. Shoutout @jake_klaris for the awesome read. https://x.com/i/article/2107213448148893696

Related

• must read if you're thinking evals for deep agents
• Without a doubt evals are the biggest blocker in most agent development pipelines. We’re going to be changing that for everyone in LangSmith…
• this is a practical guide (+ updated skill) on how we use real world data like traces + human feedback to make synthetic environments + eval…
• the first step to building evals is often times figuring out what you want to eval (sounds simple, in practice it has a lot of nuance)
• Improving agents The old way: Manually reading traces, looking for patterns, writing evals, and creating fixes. The better way: Letting Lang…

Source: Harrison Chase (X) | 2026-10-05