this is a practical guide (+ updated skill) on how we use real world data like traces + human feedback to make synthetic environments + eval…

this is a practical guide (+ updated skill) on how we use real world data like traces + human feedback to make synthetic environments + evals so we can measure, harness engineer, and post-train our agents a few main components: - World Knowledge gathering of the systems our agent will interact with - Using Specs to coordinate what will be built —> World Spec & Task Spec - Human-Agent collaboration to iteratively edit a Spec and infuse human feedback - An execution pipeline that takes an agreed on Task Spec and generates an environment + task - Fixing design flaws in the task and environment by actually running agents in the environment & mining the verifier scoring + traces @harrison_chase, @nick_hollon and I spent a ton of time going back and forth on design decisions and the overall flow - how much a coding agent should prompt for human feedback - what specs do we need and what do they contain - how to create and update world knowledge this new flow is packaged in our updated eval-engineering skill. we want to help every team own their intelligence which means owning the pipeline to convert their valuable data into Tasks+Environments that improve their agent over time check out the skill and reach out if you’re thinking about this! https://www.skills.sh/langchain-ai/langchain-skills/eval-engineering https://x.com/i/article/2092117284407926786

Source: Harrison Chase (X) | 2026-08-25

Related