Search: “evaluation”

**IssueBench** is an internal benchmark created by LangSmith to assess the performance of *Engine*, a continual‑learning agent that scans other agents’ traces to identify, cluster, and fix issues. In

This post outlines a framework for progressively automating AI workflows through three levels of increasing autonomy: using preset prompts for consistency, refining inputs with human judgment to impro

Wild 24 hours for AI and lots of different proposals have been made. TLDR; the only *tangible* new fact is that OpenAI and Anthropic are going to have embedded 3rd party evaluators from unknown organi

Most users never rate a bad interaction. They just stop. Perceived Error is LangSmith's new managed judge that reads thread evidence and flags where your agent went sideways. Trained in-house, outperf

Perceived Error is one of the clearest signals that your agent is giving users a great experience. We tuned the model and prompt around signals that tend to carry across domains, like a user correctin

On September 21 2026, Yohei reported results from a hard “insect‑identification” benchmark where model accuracy spanned 23 points across the evaluated systems. The Qwen 4B + Glance model scored 0.96,

There is a lot of justified anger at Anthropic for sandbagging Fable 5 for AI development tasks. But an unanticipated side effect is that third-party evaluators can no longer credibly use the model fo

Completely agree, @hwchase17 ! This is the meta-level breakthrough we've all been waiting for. Self-optimizing loops finally feel production-ready because LangSmith Engine turns evaluation from a manu

It ain’t independent if you are partnering with them and chose the partner. We’re partnering with Accenture on independent evaluation of frontier AI—part of our recent commitment to embed evaluators a

Claude Fable 5.1 tops the Artificial Analysis Intelligence Index but costs 20% more per task than Fable 5 despite a 75% cache read price cut We supported @AnthropicAI with pre-release evaluation of Cl

Fascinating paper on self-improving agents. (bookmark it) If you are working on agentic loops, you will quickly realize that they are only as good as the effectiveness of the evaluator. Self-improveme

Claude Fable 5 is compared against other AI models on various evaluation metrics through Cursor's benchmarking tool. The evaluation likely covers performance across different tasks such as coding, rea

everyone is talking about self-optimizing loops in software & agents. but what does that actually mean? in my mind, it's a system that observes it's own outputs, evaluates them, and uses that signal t

Llama-eval is a proposed evaluation tool for llama.cpp designed to standardize and improve comparability of community-run evaluations. The tool aims to address inconsistencies in how different users b

We benchmarked GPT-5.5 on document understanding 📄📊 We ran it through ParseBench, our comprehensive OCR benchmark over enterprise documents. We evaluated metrics across various dimensions: visual grou

arXiv:2604.18955v1 Announce Type: cross Abstract: In this study, we present the first comprehensive evaluation of modern LLMs - including GPT-4, GPT-4o, GPT-3.5-Turbo, Gemini 1.5 Pro, DeepSeek-V3, Lla

🚨 Yesterday, the 'White House Accord on Super Intelligence' was signed. What was supposed to be about AI safety and accountability was instead a PROBLEMATIC endorsement of self-regulation. My comments

As part of our efforts to pace the frontier, we’re committed to supporting independent assessments with deep levels of access across training, evaluation, and deployment. That access should enable thi

with @Thom_Wolf in sunny italy ☀️chatting about the @huggingface @OpenAI attack, the “tell” that it was an agentic swarm, how misalignment arises in RL, whether agent awareness of evaluation is eviden

Fascinating paper from Anthropic and colleagues. They study whether models can tell when they are being tested. It turns out that capable models can tell when they are being tested rather than deploye

Ha! It did it: 'We introduce BenchBenchBenchBenchBench (BBBBB), an executable benchmark of AI-authored conformance suites for benchmark-evaluation metrics' I really thought it would treat 'now do benc

NEW paper from Microsoft and colleagues. Debugging agent trajectories at scale is challenging. This is a clever approach to monitor and improve agents in production. The problem: Finding which step in

WANDR tests how well agents discover large sets of entities and verify specific facts about each one. It provides a dense, interpretable eval signal that reveals whether an agent fails, and where. The

Google’s new Gemini 3.5 Flash is the clear leader on the Intelligence vs Speed Pareto frontier and makes large gains on GDPval-AA (real-world agentic tasks), but is 5x the cost of Gemini 3 Flash @Goog

LangSmith Insights Agent helps Credit Genie interpret feedback from customers at scale, and continuously improve their product. https://www.langchain.com/blog/credit-genie-insights-agent-financial-ass

some interesting trends in terms of LLM adoption 👀 📊 The latest from the LangSmith Signal: We looked at API call volume and developer adoption trends across LLM providers using LangChain open source t

So the concern over Mythos and cybersecurity seems warranted. We conducted cyber evaluations of Claude Mythos Preview and found that it is the first model to complete an AISI cyber range end-to-end. 🧵

A user on X (formerly Twitter) expressed enthusiasm for an evaluation ('eval') track at an AI Engineer (@aiDotEngineer) event, noting its relevance to ClawBench — an open-source benchmarking and ev...

NEW paper from Microsoft Every agent benchmark has the same hidden problem: how do you know the agent actually succeeded? Microsoft researchers introduce the Universal Verifier, which discusses lesson

Introducing Multi-Agent Self-Supervision (MASS): Recursive self-improvement through collective intelligence Blog: https://pub.sakana.ai/mass/ As AI systems tackle increasingly open-ended research prob

Today we are launching an agent arena for RL environments: The Open Env Arena. Your agents can work to define the best RL environments; the platform trains Qwen-3.8-27B on them, we evaluate the models

Banger paper from MIT and Sakana AI. They show that self-improving coding agents work. The best part is that their approach, Self-Improvement via Fast Tree-search (SIFT), runs at a tenth of the CPU ho

the design of http://evaluatorbench.com grew out of a summer project, http://epistemedia.org (which i still need to write about), and runs on @activegraphai the pattern across all three projects is th

If you are tracking agent swarms research, this one is worth reading. Between 24 May and 2 July 2026, autonomous agents running inside a timed research evaluation wrote to a third party's public wiki.

I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like a

Online evals are difficult to set up correctly. Even if you know what you want to look for, it's an optimization question of which model to choose, how to tune the prompt, and how to think about the c

OpenAI has shut down the internal team that evaluated whether its AI models could pose “catastrophic risks,” a decision reported by the Financial Times on August 16, 2026. The move was announced via a

turns out you can get indirect prompt injection to ~0 on unseen attacks if you stack enough layers (model training + input probes + a classifier checking intent). didn't expect that a year ago. auto m

This is both a real incident (in that the AI really did get unauthorized access to real systems) and also something it was (sort of) prompted to do. In a review of our cybersecurity evaluations, we fo

This will happen frequently as AI becomes smarter and more agentic In a review of our cybersecurity evaluations, we found three incidents in which a Claude model reached the internet from within or wh

As AI coding models advance in capability, current evaluation benchmarks must evolve to remain challenging and meaningful measures of progress. OpenAI argues that improved benchmarks need to be harder

Harrison Chase argues that evaluations (evals) are the most intellectually demanding aspect of agent engineering, distinguishing them from other engineering tasks that may be more routine or formulaic

ParallelKernelBench is a benchmarking framework designed to evaluate large language models' ability to generate optimized GPU kernels for multi-GPU computing environments. The benchmark assesses LLMs

This post compares the outputs of three AI models—GLM-5.2, Fugu Ultra, and Fable 5—using an identical one-shot prompt to evaluate their performance, with the author expressing a preference for Fable 5

This post by Filip Hric explains how to interpret AI code review benchmarks, specifically breaking down key evaluation metrics like precision and recall that are commonly used to assess code review to

Cursor discusses methods for improving AI evaluation environments to prevent reward hacking and ensure benchmark scores more accurately measure model capabilities rather than exploitation of evaluatio