Model Releases
Online evals are difficult to set up correctly. Even if you know what you want to look for, it's an optimization question of which model to …
Online evals are difficult to set up correctly. Even if you know what you want to look for, it's an optimization question of which model to choose, how to tune the prompt, and how to think about the c
Online evals are difficult to set up correctly. Even if you know what you want to look for, it's an optimization question of which model to choose, how to tune the prompt, and how to think about the cost trade off of adding more LLM spend on top of each agent run. LangSmith Tuned Evaluators makes this much easier! Introducing LangSmith Tuned Evaluators They automatically score agent behavior in production, starting with Perceived Error. Perceived Error is one of the clearest signals that your agent is giving users a helpful experience. In our benchmark, our specialized model outperformed e…
Related
- Rubrics are even more flexible than /goal You can define a custom subagent for the grading, including custom tools, prompt, and iteration li…
- I've been thinking a lot about the two different groups of evals you need in general agents/agents which handle broad tasks: 1. Benchmark ev…
- resharing this note, find it helpful given all the great open evals work + teams building vertical agents Evals are a proxy for the behavior…
- LangSmith’s new Tuned Evaluators look pretty interesting. Perceived Error can flag agent mistakes in production by picking up on user correc…
- Prompting agents to 'do better' is unreliable 🙅 Giving them a rubric, a grader, and a correction loop is much closer to how you get your ag…
Source: Harrison Chase (X) | 2026-08-18