here's how we're improving our base harness, you can apply these same lessons to hill-climbing for your application-specific harness!

LangChain's Better-Harness system, shared by Sydney Runkle, is a compound approach to iteratively improving AI agent harnesses using evaluations (evals) as a learning signal. Better agents can be built by building better harnesses, but to autonomously build a "better" harness, a strong learning signal is needed to "hill-climb" on — evals serve as that signal, with design decisions that help agents generalize instead of overfit. The system covers the full pipeline from data sourcing → experiment design → optimization → review & acceptance, including practical details on how evals are sourced, how to design against overfitting, and how traces are stored over time. The same methodology — using evals as a hill-climbing signal — can be applied by developers to improve application-specific harnesses, via an open-sourced scaffold for autonomously improving a harness using evals as a signal at each step.

Related

• Better Harness: A Recipe for Harness Hill-Climbing with Evals
• Hands on, concrete guide (with code!) for harness hill climbing with evals
• tldr > evals are the new training data. instead of updating weights, you're updating the agent harness > problem is agents are famous cheate…
• This 10-min read from @Vtrivedy10 changes how you build AI agents. Most people are stuck in the same loop; switching models when agents brea…
• a useful mental model on how teams can think about good data design to improve their models/agents: Evals ~= Training Data ~= Environments -…

Source: agents