Search: “evaluation”

Agent evaluation is moving from scoring individual function calls to judging full task completion within executable environments that track state over multi‑step tool use. Scoring occurs at two levels

Brendan Bordelon / Politico: OpenAI backs a provision of the FRONTIER Act, a bipartisan House proposal, that would require AI companies to embed outside evaluators to ensure model safety — Artificial

smevals - a small eval suite for evaluating models, prompts, and harnesses I've been working with Jesse Vincent's Prime Radiant applied AI research lab building out this evals framework to help answer

Learn how to design, calibrate, and run LLM-as-a-judge evaluators with fixed labels, human agreement checks, trace context, and Phoenix Evals. The post How to build LLM-as-a-Judge evaluators that hold

Daniel Kokotajlo / @dkokotajlo: OpenAI researcher: top models are becoming so situationally aware humans “are losing the ability to evaluate them” while humans rely more on AI to lead research — Dan S

Dario Amodei / @darioamodei: Dario Amodei says Anthropic is “unilaterally committing” to giving third-party evaluators permanent access to verify its adherence to safety measures — We Must Pace the Fr

This post combines learnings from LangChain’s work on evaluating deep agents and Anthropic’s guide to demystifying evals for AI agents into a practical guide. In this post, you will learn how to: 1) a

In this post, you will implement four Lambda-based custom code evaluators for a financial market-intelligence agent, register each with AgentCore, and run them in on-demand and online modes. You will

Aditya Soni / Reuters: Sources: Anthropic and OpenAI have pushed for substantial control over what external AI evaluators can publish, out of concern that trade secrets could leak — Anthropic and Open

Artificial Analysis: Artificial Analysis launches the Cyber Index Alliance with partners Collinear, IBM, Nvidia, and Vercel to evaluate how AI agents find and fix vulnerabilities — The Artificial Anal

OpenAI: OpenAI releases MentalHealthBench, an open benchmark to evaluate AI responses in realistic mental health conversations, developed with 80+ licensed experts — An open benchmark developed with m

Anthropic: Anthropic partners with Accenture to embed evaluators within Anthropic; they expect to invest $2B+ in building capacity in this area over the next five years — We're partnering with Accentu

Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This para

Christine Mui / Politico: California Gov. Gavin Newsom signs into law two bills, backed by Anthropic and OpenAI, regulating how outside groups evaluate AI for safety — SAN FRANCISCO — California Gov.

When building consumer-facing generative AI applications, balancing high generation quality with fast response times across diverse media types, can be challenging. KDDI, a major telecommunications ca

AI agents are only as good as the instructions and context you give them. When we launched Google Agent Skills, our goal was simple: encode Google Cloud domain knowledge into structured, open-source i

Research: Using DSPy to evaluate and improve Datasette Agent's SQL system prompts One of this morning's AIE keynotes covered dspy, which reminded me I've been meaning to see if it could help me improv

Learn how TrueFoundry AI Gateway exports OpenTelemetry traces to Arize AX so teams can trace, evaluate, and monitor production LLM and agent traffic without embedding a vendor SDK in every service. Th

This article presents a clinical automatic speech recognition workflow for generating pronunciation-aware synthetic audio, reviewing clinical terms, and evaluating recognition quality using NVIDIA age

Automated lint: 14 errors, 54840 warnings, 3 info

Automated lint: 12 errors, 54839 warnings, 3 info

The intersection of medicine and AI has led to remarkable innovations. However, developers now face the thorny challenge of building robust medical AI tools that have been tested and evaluated on dive

Vultr is evaluating NVIDIA’s Agent Safety Platform as enterprise agents move from experimentation to production and receive real autonomy over internal systems. The platform addresses trust by securin

Cristina Criddle / Financial Times: Sources: OpenAI and Anthropic staff felt blindsided by Dario Amodei's and Sam Altman's calls to slow the frontier; some fear evaluators may compromise security — St

Joseph Cox / 404 Media: Internal OpenAI docs detail contractors evaluating anonymized prompts and chats to improve the models; model training is turned on by default for consumer plans — Humans are re

Bloomberg: Amodei says pacing does not mean halting training or progress, but giving companies time to align and safeguard models and third-party evaluators time to verify — Leaders of the world's big

Katie Roof / Business Insider: Sources: Hugging Face is exploring a sale that could value it at 13B+, up from 4.5B in 2023, and has been working with a bank to evaluate bidders' interest — The AI indu

Modern AI agents increasingly rely on search infrastructure to execute complex, neuro-symbolic reasoning workflows. These workflows often compile into deeply nested, non-monotonic Boolean queries over

Together, Motorway and AWS built an end-to-end evaluation pipeline that reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection time from few hours to few minutes. The pipelin

Use Arize Skills with Kiro CLI to trace coding-agent changes, build datasets from failures, run experiments, and validate prompts before shipping. The post Kiro CLI observability: trace and evaluate a

Proactive agents that anticipate user needs and autonomously execute tasks hold great promise as digital assistants, yet the lack of realistic user simulation frameworks hinders their development. Exi

Vercel announced a significant performance improvement to its Flags SDK, enabling flag evaluations to run 10 times faster than the previous version. This enhancement improves the speed of feature flag

This guide provides criteria and best practices for assessing enterprise analytics platforms, likely covering key evaluation factors such as scalability, performance, integration capabilities, ease of

LeRobot v0.6.0 is a release from Hugging Face that introduces new capabilities for robotic learning, likely including features for imagining/generating robot behaviors, evaluating model performance, a

Marina Temkin / TechCrunch: Patronus AI, which builds simulated digital environments for evaluating AI agents, raised a 50M Series B led by Greenfield, bringing its total funding to 70M — AI agents ar

You ship an agent. It mostly works. Then the feedback loop becomes a support ticket: a user got the wrong answer, a tool call looped four times, a refund went out that shouldn’t have. Standard observa

Automated lint: 17 errors, 59669 warnings, 3 info

Automated lint: 12 errors, 53990 warnings, 3 info

A local coding agent, an in-app customer assistant, and an AI SRE triaging production logs may all use the same model class—but not the same harness, eval plan, or rollout risk. Mastra CEO Sam Bhagwat

In this post, we walk you through the Nova Sonic Test Harness, an open source framework that we built to solve both problems. It serves as a rapid iteration tool for tuning system prompts and tool con

Autoheal AI Inc., an artificial intelligence-native platform engineering startup that’s trying to pioneer the concept of “self-improving software factories,” said today it has raised 7.9 million in se

Skills let you encode domain-specific procedures as reusable, portable instructions for agents, but a fluent answer doesn't prove the agent picked the right skill or followed it. Learn how to measure

NVIDIA SkillEvaluator is an open‑source framework that measures how pre‑verified “skills” improve AI agent performance through static checks, distinctiveness analysis, and live task runs in isolated e

At ChatGPT scale, user frustration arrives as support tickets, ratings, social posts, and corrections buried inside conversations. OpenAI built a feedback system that can find the pattern behind a com

Agent evals are repeatable tests that score whether AI agents completed a task correctly. Learn how to design rubrics, test suites, and trace-based evals that catch failures and prevent reward hacking

If you’re building visual shopping, image or document understanding, or chart analysis, you need a way to verify whether your model’s response is actually grounded in the source image. A text-only eva