Search: “benchmark”

On benchmarking long-context agentic instruction following. Agent benchmarks mostly reward reaching the answer. This new benchmark measures whether the agent reached it the permitted way, which is the

Recently, Datalab released OmniExtractBench, a benchmark that combines several benchmarks (our own ExtractBench, LongExtractBench, other vendors) to measure structured document quality. The benchmark

Epoch continues to do some of the best public benchmarking work on AI. This is helpful (and tells us how bad the state of benchmarking is, and how terrible some of our favorite benchmarks are) Introdu

Ha! It did it: 'We introduce BenchBenchBenchBenchBench (BBBBB), an executable benchmark of AI-authored conformance suites for benchmark-evaluation metrics' I really thought it would treat 'now do benc

We benchmarked GPT-5.5 on document understanding 📄📊 We ran it through ParseBench, our comprehensive OCR benchmark over enterprise documents. We evaluated metrics across various dimensions: visual grou

ParseBench is the first benchmark to include VLM chart understanding 📊📈📉 over enterprise documents. 🟠 Existing benchmarks (ChartQA, ChartXiv) test over charts specifically and not the chart's inclusio

Gemini 3.8 Flash might be maxxed on other benchmarks, but not quite on document parsing. We benchmarked it on ParseBench. It does slightly better on tables + content faithfulness compared to Gemini 3.

As the benchmarks that test frontier AI on get more complex, we are losing one of the most important aspects of benchmarking: comparisons to humans Validated benchmarks need to have human (ideally mul

Benchmarking AI systems is a continual process that co-evolves with the models. New benchmarks challenge AI capabilities with emerging questions to shape the directions and feedback signal of the rese

ParseBench: A benchmark for document parsing agents @llama_index just shipped a benchmark with 2k verified pages for real enterprise documents. Benchmarks are the major underrated component in the ML

As a joke I prompted Codex 'Build and run BenchBench, a benchmark of now good ai is at creating benchmarks. then figure out what benchbenchbench is and run that. and then write benchbenchbench up as a

Peak performance across hard benchmarks: • Best or joint-best on 5/8 benchmarks (DeepSWE, Chartography, Toolathon, GDP.pdf, SWEFish) • Chartography: 48.3 (outperforming Opus 5 & Fable 5) • DeepSWE: 74

CancerBench: the frontier model cancer cure benchmark. AI lab CEOs keep talking about curing cancer, so I made a benchmark. One metric: how many types of cancer has your model cured? All models are cu

Introducing SWE-Together: a multi-turn benchmark built from real user–agent coding sessions. Coding agents are often benchmarked like exam-takers: given the full spec up front, then graded on the fina

Harrison Chase expresses enthusiasm about collaborating with Harvey on their Long Horizon Legal Agent Benchmark, highlighting the value of developing industry-specific benchmarks for AI evaluation. Th

I think Artificial Analysis does a good job overall and provides transparency in benchmarking, but GDPval-AA is not a good benchmark and needs to stop being reported. It is Gemini 3.1 judging other mo

.@Kimi_Moonshot benchmarked K3 endpoints across major inference providers. Together AI leads or ties for #1 on 3 of 4 benchmarks: OCRBench, MMMU Pro Vision, and DeepSWE. Open models like Kimi K3 have

Open models should compete on cost and specialization, not frontier benchmarks @natolambert puts it well: the right benchmark is savings in compute and time, especially for repetitive agent tasks deep

A challenge with AI regulation and vetting is how bad our benchmarks of AI model performance and risks are. There is no benchmark for risks and red-teaming requires experiments from dedicated speciali

We comprehensively benchmarked Opus 4.7 on document understanding. We evaluated it through ParseBench - our comprehensive OCR benchmark for enterprise documents where we evaluate tables, text, charts,

ARC-AGI-3 has the lowest human bar of any AI benchmark out there. Almost all benchmarks require specialized knowledge that make them inaccessible to 99%+ of humans (like, say SWE-Bench). ARC-AGI-3 is

Document OCR benchmarks are still an open problem Existing document OCR benchmarks are either too narrowly focused on a specific type (e.g. FinTabNet, ChartQA), or on documents that aren’t reflective

Benchmarked @DJLougen ’s Ornstein-27B-v2 Q6_K on my RTX 3090 using hermes-bench, my new open-source benchmarking UI for local LLMs and Hermes agents. Ornstein is a Qwen 3.5 27B fine-tune trained on re

We benchmarked GPT-6 Astra on hard document parsing and extraction tasks. It is the best frontier model for one-shotting document extraction over short and medium-sized documents: 📊 97.2% over short d

Interesting new research from IBM. If you pick models from benchmark deltas, some of that delta belongs to the phrasing rather than the model. BenchDrift generates meaning-preserving variations of ben

Standard RL benchmarks are episodic and stationary, so they don't capture the the characteristics of real-world deployment. Morpheus is a new benchmark for continual learning that provides persistent

The state of public AI benchmarking is dire and is undermining our ability to understand how good AI is now. Most famous measures are maxed out, and, as this paper shows, the non-saturated benchmarks

This is a really neat benchmark. It's super interesting that Astra starts strong and holds the lead for up to 19 hours, but Fable 5.1 catches up in the final hours. Not entirely clear why. Different m

The benchmark for non-verifiable domains is often the opinions of humans. That is how we determine whether writing or an idea or a pitch is good in the real world And we know how to measure & benchmar

Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but su

We’re open sourcing WANDR. WANDR is an internal benchmark we built and used for building deep and wide research capabilities inside Perplexity Computer. https://research.perplexity.ai/articles/wandr-b

Huge release from @odysseyml. Odyssey-3 Pro sets a new top score on Physics-IQ Verified, a benchmark that asks models to continue videos of real physics experiments. The robotics results stood out to

About a year ago, before it was on anyone's radar, we began exploring the idea of a benchmark for open-ended invention. Since then, we've developed several promising directions that will serve as the

It's well known that agents hack benchmark rewards. The usual response is a patch for each task that gets exploited. In a study of 456 adjudicated trajectories from more than 31,000 public agent runs,

We've benchmarked all the latest frontier models on hard document extraction tasks (with Fable 5.1 and Gemini 3.8 Flash coming soon!). ExtractBench is now live on @kaggle . 5.6 Sol leads the pack, fol

Big shoutout to @perplexity_ai for properly benchmarking document understanding in their Portable Computer release 🔥 They took a subset of our ParseBench benchmark (https://www.parsebench.ai/) and mea

Introducing 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗕𝗲𝗻𝗰𝗵: the most comprehensive benchmark for information extraction from complex enterprise documents. Our applied research team tested: 14 systems — frontier VLMs, coding agents, ex

Let's talk parsing charts 📊📈. Last week we released ParseBench, the first document OCR benchmark for AI agents. New in ParseBench: ChartDataPointMatch. Most document look at a chart and OCR the captio

Here's a benchmark that uses Lilypond https://x.com/aug5thmusic/status/2096030719156089029 GPT-6 Astra has the best result yet on the Bach Benchmark. Its chorale contains no voice-leading errors, and

What is a way to test human-level generality in artificial intelligence? 'The meta-benchmark of being able to pass ARC-AGI-(n+1) immediately upon release' @Jabaluck @DAcemogluMIT The meta-benchmark of

Jensen Huang says that AGI has been achieved. I strongly disagree. GPT-6 performs really well on benchmarks, including ARC-AGI-3. But benchmark performance is not the same as general intelligence. GPT

ExtractBench is one of the most comprehensive benchmarks for real-world document extraction. ✅ It covers 4869 pages, across 67 document types, spanning 8 real-world domains: finance, energy, gov, auto

NEW paper from Microsoft Every agent benchmark has the same hidden problem: how do you know the agent actually succeeded? Microsoft researchers introduce the Universal Verifier, which discusses lesson

Sebastian Raschka (rasbt) used Randal Olson's Tufte Test tool, developed by Goodeye Labs, to improve the visual quality of a machine learning benchmark plot. The Tufte Test encodes seven of Tufte'...

MistralAI announced that its model **Mistral Large 4** ranked as the #1 open‑source (OSS) model on **Harvey’s Legal Agent Benchmark (LAB)**, an evaluation benchmark for legal AI agents. The tweet post

BREAKING: Grok 4.7 dominates the Legal Agent Benchmark. It scored 19.6% on realistic, long-term legal tasks, beating every other model tested. That is nearly 3× Fable 5.1. Another major win for Grok 4