my fave question, talked about this coding agent Eval+Improvement loop infra + UX in my AIE talk yesterday! biased but LangSmith is the best…

my fave question, talked about this coding agent Eval+Improvement loop infra + UX in my AIE talk yesterday! biased but LangSmith is the best spot to Eval + continuously improve your coding agents, and we want to make it better so would love any feedback :) we eval all of our coding agents there --> supports Codex, Claude Code, OpenCode, Deep Agents, Pi, etc all into Tracing, sandbox infra for running evals, metrics + datasets for storing everything, and imo the hardest parts of doing coding agent evals are: 1. having infra to easily store, update, run, and share coding agent evals 2. building a clear understanding of agent behavior & failure modes across all of the rollouts. you can use a coding agent + the langsmith-cli to look through all the data or if teams want a managed, they can use LangSmith Engine to read every single trace from every eval, see what went wrong, prepare a report for you, and propose new changes directly in the code to fix issues 3. build new and better evals for your coding tasks. there's many ways to do this, but we find that looking through existing failure modes from eval runs and prod is a really good grounded way to measure where agents lack today and turn that data into we recently launched a LangSmith x @harborframework integration to double down on making it super easy for LangSmith users to improve their coding agents over time with built-in infra for running large-scale containerized evals so you can fully reproduce your tasks and read all of the traces https://www.langchain.com/blog/unified-stack-for-evaluating-agents i think most evals in the future will be shaped as environments and letting agents do real work in them, because agents are doing way more complex things and we need the eval shape to mirror how they work with us if there's any experience you're looking for would love to chat we have a ton of work underway on making each part of coding agent evaluation + improvement easier over time (as teams eval their coding agents for months and years) as an aside --> evals are literally the training data for agents. the behaviors that we measure and reward in the evals directly get transferred as model/agent behavior as we hill-climb them. so i think there's nothing more important than making it easy to build good evals over time :) @Vtrivedy10 do you know of any eval platforms that work with coding agents? Unless I'm blind, everything looks like it's product-agent focused. I need something that will work with coding agents on complex R&D tasks. (currently building my own so we can properly eval our harness)

Source: Harrison Chase (X) | 2026-07-02