This 10-min read from @Vtrivedy10 changes how you build AI agents. Most people are stuck in the same loop; switching models when agents brea…

This 10-min read from @Vtrivedy10 changes how you build AI agents. Most people are stuck in the same loop; switching models when agents break. The reframe: evals are the training data for your harness. 1. If you want the first-principles foundation behind all of this, here's the 2-hour Stanford lecture on LLM evaluation. 2. The recipe: > Source evals from production traces; every failure is a training example > Tag by behaviour; tool selection, reasoning, followup quality > Split into optimisation and holdout sets; agents will overfit without this > One change at a time; diagnose, edit, validate, repeat Keep this for later. You'll need it when your agent breaks at 11pm and you don't know why. Media

tldr > evals are the new training data. instead of updating weights, you're updating the agent harness > problem is agents are famous cheate…
a useful mental model on how teams can think about good data design to improve their models/agents: Evals ~= Training Data ~= Environments -…
Agents will cheat your evals if you let them. @Vtrivedy10 wrote a great article on how we keep them from overfitting
Self-improving agents isn’t a single algorithm - it’s a systems engineering problem involving: - eval data curation + maintenance - experime…
New Guide: Incorporating human judgment in the agent improvement loop Building agents is hard. Everyone talks about the code. What gets less…
was really fun to sit down with @isidoremiller for this one! he has a bunch of hot takes on agents and evals that you're going to want to he…

Source: Harrison Chase (X) | 2026-04-09

Related