a useful mental model on how teams can think about good data design to improve their models/agents: Evals ~= Training Data ~= Environments -…

a useful mental model on how teams can think about good data design to improve their models/agents: Evals ~= Training Data ~= Environments - in Classical Deep Learning, we learn from each training example to compute an edit to the weights and “improve our model”. Bad Data = Bad Model, we spent tons of hours (and money) on data curation like labeling data and sourcing diverse data. - Evals are the currency of improvement for agents. We design tasks for agents across domains and capabilities like reasoning and retrieval. Evals encode the behaviors we want agents to follow in production. Every eval nudges the agent definition to make sure it passes. Evals also provide a signal for the agent designer (human or another agent) on how to adjust the agent definition to make that eval pass. The same behaviors encoded in the eval set will appear in the agent - “Environments rhyme with evals” - an environment is a place for agents to do work and then it specifies a reward by evaluating that work. this might be a rubric of things the agent did. Often environments are synonymous with RL, but many more gradient-free methods (hill climb the harness) will build environments for the agent to work in and another agent will review the trace to see what to adjust. For example if you’re testing an agent’s ability to navigate your app and do work, you’ll want to build that environment for your agent to test it on evals - I bet evals will predominantly shift to be sandboxed environments as agents take on more complex work

Source: Harrison Chase (X) | 2026-04-09

Related