Search: “evaluation”

Perplexity's reward design system prioritizes correctness as a foundational requirement, then evaluates user preference and efficiency only when answers meet correctness standards. This hierarchical a

Today we’re introducing two big steps for health at OpenAI: - ChatGPT for Clinicians, a free version of ChatGPT designed for clinical work - HealthBench Professional, a new benchmark to evaluate real

VideoGameBench is a new benchmark launched on Antim Labs, created by a1zhang, Thomas L. Griffiths, Karthik R. N, and Ofir Press. The benchmark likely evaluates AI model performance on video game-relat

Agent evals are drifting away from production reality. Most benchmarks use clean tasks, well-specified requirements, deterministic metrics, and retrospective curation. Production work is messier, with

5/5 The takeaway: If your agent relies on an LLM judge for selection accuracy, measuring code quality isn’t enough; you need a measure of the model's inductive bias toward the 'fingerprint' of a gold

Let's talk parsing tables. Two days ago we launched ParseBench,the first document OCR benchmark built for AI agents. This deep dive breaks down TableRecordMatch (GTRM), our metric for evaluating compl

ParseBench is the most comprehensive OCR benchmark for real-world enterprise documents: financial filings, contracts, insurance documents, and more. We evaluate across 5 dimensions that are present am

EinsteinArena is an open-source benchmarking platform developed by Together AI designed to evaluate and rank AI models, with a live public leaderboard tracking model performance. The project welcomes

What does it actually take to make agents better over time? A system that starts with a trace. You capture traces of agent behavior, enrich them with evaluations and human feedback, identify what’s fa

LangChain's **Better-Harness** system, shared by Sydney Runkle, is a compound approach to iteratively improving AI agent harnesses using evaluations (evals) as a learning signal. Better agents can...

NEW paper from Google on multi-agent research agents. It's one of the first systems that handles end-to-end LaTeX generation, targeted literature reviews, and conceptual diagrams as a decoupled, stand

The new model from Meta is already looking like a disappointment: overoptimized for public benchmark numbers at the detriment of everything else. Knowing how to evaluate models in a way that correlate

Join the ARC Prize team -- help us build ARC-AGI-4 and ARC-AGI-5 Platform Engineer - Benchmark Lead ARC Prize Foundation is hiring a senior engineer to build our benchmark platform * Expand ARC-AGI-3

Benchmarks are becoming the core competency of any company. Our company mission is to build the world’s best models for document processing. This also means that we have to build the world’s best benc

Much of the valuable information in agent traces isn’t directly filterable with existing techniques like full text search or metadata filtering. AI Search in LangSmith closes that gap. You describe wh

Zai recently launched a domain‑specific language model called **GLM Legal**. According to a tweet posted at 4:13 PM on Oct 12, 2026, GLM Legal became the leading open‑weight model on both **Harvey’s L

New paper from the team at Sakana AI: Recursive Self-Improvement through Multi-Agent Self-Supervision (MASS) How do we keep improving AI when the tasks become too complex for humans to reliably evalua

Super interesting NVIDIA paper on choosing base models for coding agents. It's actually a clever way to rank base checkpoints by how well each one is likely to do as a coding agent after post-training

Gmail. Google Sheets. Slack. Salesforce. i.e. the four horsemen of Monday. On AutomationBench, Mistral Large 4 took on 657 business workflows across simulated workplace apps, all four included, finish

AI is driving the biggest infrastructure build-out in history, with @Google alone forecasting $200B of capex in 2026E. What are the real constraints on a system at this scale? Amin Vahdat is the chief

Malware reverse-engineering: solving an out-of-distribution investigation task. When faced with an unknown binary, Mistral Large 4 reverse-engineers it end-to-end. In this case, it concludes the sampl

On October 3, 2026 Yohei Nakajima posted a brief tweet praising the use of multiple AI coding assistants—Codex, Claude, Gemini, and Grok Bot—with a musical note emoji. The message also contains a Java

We’re also reopening Pro 200 subscriptions, with continued access to frontier models like Astra, including our new GPT-6.1 Sol model which brings near-Astra capabilities to a model you can use every d

every agent will need a sandbox and the harness should live outside it - separate the brains from the hands great to see nvidia open sourcing tools here, agree with Andrew this is part of the harness

Only an open ecosystem can guarantee the safety of AI Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform, bringing together OpenShell and Sentry. Artificial in

Personal agents gone wrong. As we embrace more personal agents to carry out personalized tasks in the real world, interesting dynamics and behaviors will emerge. I think personal agents as they stand

Cool paper on catching alignment failures with Jev. The ideas is to ask Jev one generic yes/no question about a model's response, and use its probability as a score. With no extra training, that score

Introducing 'Scaling In-Context Imitation Learning' (SAIL) to be presented at #IROS2026. This work is a collaboration between Sakana AI and the University of Tokyo. Blog: https://pub.sakana.ai/sail Wh

The era of omni agents is upon us. This is a great report by the Qwen Team on their omni-modal agents. They present Qwen3.8-Omni-Flash, a natively multimodal model trained for long-horizon agent tasks

coding agents are moving fast from prototype to production. the infrastructure question is what's left. join @parthsareen from @ollama and @zainhas Hasan from Together AI at @AIconference for a breako

jev as a Judge proves to be a cheaper and more precise alternative to LLM as a judge for online evals. great guide from Sean and Daniel on why Jev is great for evals, an experiment vs other LLMs, and

They don’t have a fucking clue. Not even a theory about how make their stuff safe. The simple truth: not even Anthropic has a plan for how to make AI safe in advance. There is no theory. I've asked! O

Schmidhuber was building recursive self-improving systems back in 1987. His new post covers four decades of RSI, from meta-evolution and self-modifying policies to the Gödel Machine and modern LLM age

Banger report from Salesforce. Pretty interesting to see more of these custom enterprise models. Salesforce trained the enterprise agent model from the same files it uses to configure agents. Koa star

Nice work from Adobe. It's standard practice to store a fixed reference answer for every eval case. When the underlying data changes daily, that stored answer goes stale. Adobe researchers write each

Which AI models are most likely to cheat when given the chance? To find out, we built CheatBench [https://cheatbench.ai] : a benchmark that tests whether agents attempt to cheat when given difficult t

Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent i

What does it mean to pace the frontier? Over the last month, @random_walker and I have analyzed the loss-of-control incidents at AI companies to understand what technical and policy interventions can

1) The HuggingFace attack was a felony under the Computer Fraud and Abuse Act. So were Anthropic’s Claude gaining “unauthorized access to the production infrastructure of three different organization(

For context. OpenAI turned to METR for independent investigation of the hugging face incident and Anthropic is inviting them in as a third-party. (Industry self-regulation is not always the best path

It's now clear that: - alignment is critical to making AI safe - alignment won't be solved behind the closed doors of a handful of frontier labs So we're taking two steps: - launching the Open Alignme

It's now clear that alignment is critical and won't be solved behind the closed doors of a handful of frontier labs. So today we're launching the Open Alignment Initiative, led by @Thom_Wolf @huggingf

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’l

We work extremely hard to align our models as well as we can but this is a very hard job. I am particularly excited about two things this proposal calls for that will hopefully make it somewhat easier

Another brilliant paper from Meta. This one is worth reading if you work on production-grade ranking or recommendation systems. Meta deployed an autonomous agent that runs the ML iteration cycle acros

Recommended read. Interaction horizon scheduling is an underexplored control problem in agentic RL This paper from the Qwen team takes a closer look at the problem. Scaling the maximum number of envir