Knowledge catalogue
Fable 5.1 didn't come with document OCR benchmarks, so we benchmarked it on ParseBench: a comprehensive parsing benchmark that's comprised of 2k+ real-world documents across finance, legal, insurance,
Epoch AI has launched “Benchmark Reviews,” an initiative to evaluate existing AI benchmark datasets. The program initially audits 15 benchmarks, categorizing them as 4 verified, 9 flawed, and 2 lackin
On benchmarking long-context agentic instruction following. Agent benchmarks mostly reward reaching the answer. This new benchmark measures whether the agent reached it the permitted way, which is the
Recently, Datalab released OmniExtractBench, a benchmark that combines several benchmarks (our own ExtractBench, LongExtractBench, other vendors) to measure structured document quality. The benchmark
Epoch continues to do some of the best public benchmarking work on AI. This is helpful (and tells us how bad the state of benchmarking is, and how terrible some of our favorite benchmarks are) Introdu
Ha! It did it: 'We introduce BenchBenchBenchBenchBench (BBBBB), an executable benchmark of AI-authored conformance suites for benchmark-evaluation metrics' I really thought it would treat 'now do benc
We benchmarked GPT-5.5 on document understanding 📄📊 We ran it through ParseBench, our comprehensive OCR benchmark over enterprise documents. We evaluated metrics across various dimensions: visual grou
ParseBench is the first benchmark to include VLM chart understanding 📊📈📉 over enterprise documents. 🟠 Existing benchmarks (ChartQA, ChartXiv) test over charts specifically and not the chart's inclusio
Let's talk parsing charts 📊📈. Last week we released ParseBench, the first document OCR benchmark for AI agents. New in ParseBench: ChartDataPointMatch. Most document look at a chart and OCR the captio
I find that open weights models over-perform on benchmarks compared to actual real-world usage, and Kimi feels like no exception. For example, a small amount of use will show that Kimi is not as good
Gemini 3.8 Flash might be maxxed on other benchmarks, but not quite on document parsing. We benchmarked it on ParseBench. It does slightly better on tables + content faithfulness compared to Gemini 3.
As the benchmarks that test frontier AI on get more complex, we are losing one of the most important aspects of benchmarking: comparisons to humans Validated benchmarks need to have human (ideally mul
Benchmarking AI systems is a continual process that co-evolves with the models. New benchmarks challenge AI capabilities with emerging questions to shape the directions and feedback signal of the rese
We've benchmarked all the latest frontier models on hard document extraction tasks (with Fable 5.1 and Gemini 3.8 Flash coming soon!). ExtractBench is now live on @kaggle . 5.6 Sol leads the pack, fol
I trust these vibe checks more than I trust most of the benchmark results GPT-6 astra seems more powerful for e2e automation and better at writing (a massively underrated gap for these models). But it
ParseBench: A benchmark for document parsing agents @llama_index just shipped a benchmark with 2k verified pages for real enterprise documents. Benchmarks are the major underrated component in the ML
As a joke I prompted Codex 'Build and run BenchBench, a benchmark of now good ai is at creating benchmarks. then figure out what benchbenchbench is and run that. and then write benchbenchbench up as a
We benchmarked Opus 5 comprehensively on document understanding through ParseBench. It is roughly on par with Opus 4.8 - it does a few points worse on dense tables, but does slightly better on parsing
Peak performance across hard benchmarks: • Best or joint-best on 5/8 benchmarks (DeepSWE, Chartography, Toolathon, GDP.pdf, SWEFish) • Chartography: 48.3 (outperforming Opus 5 & Fable 5) • DeepSWE: 74
About a year ago, before it was on anyone's radar, we began exploring the idea of a benchmark for open-ended invention. Since then, we've developed several promising directions that will serve as the
It's well known that agents hack benchmark rewards. The usual response is a patch for each task that gets exploited. In a study of 456 adjudicated trajectories from more than 31,000 public agent runs,
CancerBench: the frontier model cancer cure benchmark. AI lab CEOs keep talking about curing cancer, so I made a benchmark. One metric: how many types of cancer has your model cured? All models are cu
This is a really neat benchmark. It's super interesting that Astra starts strong and holds the lead for up to 19 hours, but Fable 5.1 catches up in the final hours. Not entirely clear why. Different m
We're introducing Q2D-Web (Query2Doc-Web), a benchmark and public leaderboard for evaluating retrieval in agentic RAG systems. Q2D-Web tests how embedding models perform on large-scale web search usin
Introducing SWE-Together: a multi-turn benchmark built from real user–agent coding sessions. Coding agents are often benchmarked like exam-takers: given the full spec up front, then graded on the fina
Harrison Chase expresses enthusiasm about collaborating with Harvey on their Long Horizon Legal Agent Benchmark, highlighting the value of developing industry-specific benchmarks for AI evaluation. Th
I think Artificial Analysis does a good job overall and provides transparency in benchmarking, but GDPval-AA is not a good benchmark and needs to stop being reported. It is Gemini 3.1 judging other mo
.@Kimi_Moonshot benchmarked K3 endpoints across major inference providers. Together AI leads or ties for #1 on 3 of 4 benchmarks: OCRBench, MMMU Pro Vision, and DeepSWE. Open models like Kimi K3 have
Open models should compete on cost and specialization, not frontier benchmarks @natolambert puts it well: the right benchmark is savings in compute and time, especially for repetitive agent tasks deep
A challenge with AI regulation and vetting is how bad our benchmarks of AI model performance and risks are. There is no benchmark for risks and red-teaming requires experiments from dedicated speciali
We comprehensively benchmarked Opus 4.7 on document understanding. We evaluated it through ParseBench - our comprehensive OCR benchmark for enterprise documents where we evaluate tables, text, charts,
ARC-AGI-3 has the lowest human bar of any AI benchmark out there. Almost all benchmarks require specialized knowledge that make them inaccessible to 99%+ of humans (like, say SWE-Bench). ARC-AGI-3 is
Document OCR benchmarks are still an open problem Existing document OCR benchmarks are either too narrowly focused on a specific type (e.g. FinTabNet, ChartQA), or on documents that aren’t reflective
Benchmarked @DJLougen ’s Ornstein-27B-v2 Q6_K on my RTX 3090 using hermes-bench, my new open-source benchmarking UI for local LLMs and Hermes agents. Ornstein is a Qwen 3.5 27B fine-tune trained on re
We benchmarked GPT-6 Astra on hard document parsing and extraction tasks. It is the best frontier model for one-shotting document extraction over short and medium-sized documents: 📊 97.2% over short d
Interesting new research from IBM. If you pick models from benchmark deltas, some of that delta belongs to the phrasing rather than the model. BenchDrift generates meaning-preserving variations of ben
Standard RL benchmarks are episodic and stationary, so they don't capture the the characteristics of real-world deployment. Morpheus is a new benchmark for continual learning that provides persistent
The state of public AI benchmarking is dire and is undermining our ability to understand how good AI is now. Most famous measures are maxed out, and, as this paper shows, the non-saturated benchmarks
The benchmark for non-verifiable domains is often the opinions of humans. That is how we determine whether writing or an idea or a pitch is good in the real world And we know how to measure & benchmar
Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but su
Introducing 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗕𝗲𝗻𝗰𝗵: the most comprehensive benchmark for information extraction from complex enterprise documents. Our applied research team tested: 14 systems — frontier VLMs, coding agents, ex
ExtractBench is one of the most comprehensive benchmarks for real-world document extraction. ✅ It covers 4869 pages, across 67 document types, spanning 8 real-world domains: finance, energy, gov, auto
We’re open sourcing WANDR. WANDR is an internal benchmark we built and used for building deep and wide research capabilities inside Perplexity Computer. https://research.perplexity.ai/articles/wandr-b
Huge release from @odysseyml. Odyssey-3 Pro sets a new top score on Physics-IQ Verified, a benchmark that asks models to continue videos of real physics experiments. The robotics results stood out to
Big shoutout to @perplexity_ai for properly benchmarking document understanding in their Portable Computer release 🔥 They took a subset of our ParseBench benchmark (https://www.parsebench.ai/) and mea
We benchmarked 20+ open-weight models on easy-to-hard document extraction tasks through ExtractBench. The results are all available on @huggingface 🤗 ExtractBench is a schema-guided extraction benchma
Here's a benchmark that uses Lilypond https://x.com/aug5thmusic/status/2096030719156089029 GPT-6 Astra has the best result yet on the Bach Benchmark. Its chorale contains no voice-leading errors, and
What is a way to test human-level generality in artificial intelligence? 'The meta-benchmark of being able to pass ARC-AGI-(n+1) immediately upon release' @Jabaluck @DAcemogluMIT The meta-benchmark of
Jensen Huang says that AGI has been achieved. I strongly disagree. GPT-6 performs really well on benchmarks, including ARC-AGI-3. But benchmark performance is not the same as general intelligence. GPT
NEW paper from Microsoft Every agent benchmark has the same hidden problem: how do you know the agent actually succeeded? Microsoft researchers introduce the Universal Verifier, which discusses lesson
Sebastian Raschka (rasbt) used Randal Olson's Tufte Test tool, developed by Goodeye Labs, to improve the visual quality of a machine learning benchmark plot. The Tufte Test encodes seven of Tufte'...
MistralAI announced that its model **Mistral Large 4** ranked as the #1 open‑source (OSS) model on **Harvey’s Legal Agent Benchmark (LAB)**, an evaluation benchmark for legal AI agents. The tweet post
BREAKING: Grok 4.7 dominates the Legal Agent Benchmark. It scored 19.6% on realistic, long-term legal tasks, beating every other model tested. That is nearly 3× Fable 5.1. Another major win for Grok 4
GLM-5.3-Flash is live on Fireworks on day… 2 Why? Because we take quality very seriously. We found a benchmark discrepancy we couldn’t explain, so we delayed the launch to investigate. Day 0 (Wed): we
On August 7, 2026 Ethan Mollick tweeted that “every remaining good AI benchmark score has an implied asterisk next to it which reads: * could be significantly higher with a better harness.” The commen
Introducing AI IQ Bio: the most comprehensive set of biotech benchmarks in the world ...& Bio IQ: the most comprehensive 'biotech capabilities index' ever produced Benchmark sources include benchmarks
There are a lot of coding and reasoning benchmarks for AI agents, but not a lot for document understanding - which is a prerequisite for all downstream knowledge work. We released ParseBench ~a month
Recommended benchmark. I expect voice to become one of the main ways people interact with robots and physical AI. That makes speech recognition in real conversations much more important than it seems
Claude Opus 5.5 is now available as the Standard effort level in Perplexity Computer. On our WANDR benchmark, it scored 0.610 at $4.13 per task, slightly outperforming Claude Fable 5.1 while costing 6
Our general-purpose coding agent just scored 100% on the ARC-AGI-3 interactive reasoning benchmark. NVIDIA AVO completed all 183 levels across all 25 public environments, figuring out what to do with