Agents
tldr > evals are the new training data. instead of updating weights, you're updating the agent harness > problem is agents are famous cheate…
tldr > evals are the new training data. instead of updating weights, you're updating the agent harness > problem is agents are famous cheaters. they will reward-hack your evals and overfit just to mak
tldr > evals are the new training data. instead of updating weights, you're updating the agent harness > problem is agents are famous cheaters. they will reward-hack your evals and overfit just to make the score go up > solution is treat evals like real ml. you need strict train/test splits and holdout sets to force actual generalization > quality over quantity. a few hand-curated, highly-tagged evals easily beat thousands of noisy ones > the better-harness loop is mine prod traces for failures -> turn into evals -> auto-tweak prompts & tools -> validate > the ultimate flywheel is more prod usage -> more failure traces -> more evals -> continuously self-improving agents
Related
- a useful mental model on how teams can think about good data design to improve their models/agents: Evals ~= Training Data ~= Environments -…
- Self-improving agents isn’t a single algorithm - it’s a systems engineering problem involving: - eval data curation + maintenance - experime…
- What does it actually take to make agents better over time? A system that starts with a trace. You capture traces of agent behavior, enrich …
- New Guide: Incorporating human judgment in the agent improvement loop Building agents is hard. Everyone talks about the code. What gets less…
- here's how we're improving our base harness, you can apply these same lessons to hill-climbing for your application-specific harness!
- Another banger article from the @LangChain team! Harness evolution combined with specialist local models will be the way forward undoubtedly…
Source: Harrison Chase (X) | 2026-04-10