Search: “benchmark”

Jerry Liu announces that the ParseBench leaderboard is automatically updated on Kaggle, providing a continuously maintained benchmark for parsing performance metrics. The leaderboard is hosted under t

The tweet from @NousResearch links to a Windows Central article written by Zac Bowden that provides an in‑depth review and benchmark analysis of Nvidia’s RTX Spark. Posted on Oct 16, 2026, the review

ParseBench is now also available on Papers with Code! Find it here: https://paperswithcode.co/benchmark/parsebench We benchmarked Mistral OCR against other frontier and open-weight models on ParseBenc

really great thread/benchmark for common sense in robotics and a reminder that we are not actually close to AGI GPT-6 Astra attempted harmful actions 97% of the time when it was asked to stab a human-

Companion to the Qwen 3.8 Release Megathread. Compiled from ~2,000 posts scanned across both subs, with deep reads of the 45 highest-signal threads (560 posts and comments), Aug 15–22, 2026, plus inde

We wrote a 36-page ArXiv whitepaper on ExtractBench 🧑‍🔬 , our effort to create the most comprehensive, schema-guided, real-world document extraction benchmark. It’s extremely detailed and covers every

Everyone who think vision is solved should look at some enterprise documents We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of

Unlike other players in the OCR space, we don't block our competitors from accessing our platform. We're also fully committed towards open, reproducible benchmarks for document parsing and extraction

Join the ARC Prize team -- help us build ARC-AGI-4 and ARC-AGI-5 Platform Engineer - Benchmark Lead ARC Prize Foundation is hiring a senior engineer to build our benchmark platform * Expand ARC-AGI-3

We think a lot about the distribution of all real-world documents, particularly for real-world use cases. We think about the distribution in terms of page length, field length, domain, and task type.

There's a lot of real-world documents that are scanned, rotated, handwritten, or some combination of any of these elements. This week we created a comprehensive document extraction benchmark that cont

This week we launched the world's most accurate document extraction agent over real-world documents. Introducing LlamaExtract Agentic Plus 💫 . It is a complete document extraction model+harness engine

Document extraction is an arbitrarily complex reasoning problem for both frontier models and humans. ✅ The schema encodes essentially hundreds of interrelated reasoning tasks ✅ A PDF can be hundreds o

just ran this on my phone (17 pro) Every on-device AI benchmark you've seen was measured on someone else's phone. So we stopped posting numbers and shipped the benchmark instead. Built into the RunAny

VideoGameBench is a new benchmark launched on Antim Labs, created by a1zhang, Thomas L. Griffiths, Karthik R. N, and Ofir Press. The benchmark likely evaluates AI model performance on video game-relat

ARC-AGI-3 is a new benchmark iteration designed so that any intelligent human applying genuine effort should achieve greater than 90% accuracy, maintaining the benchmark's core principle that tasks mu

LlamaIndex 发布了一个面向 AI Agent 的文档解析评测基准「ParseBench」 @llama_index 100 种解析方法参与评测（vlm - 67、专用解析器 - 33），2000 页真实企业文档（人工两轮校验），169011 条确定性规则，完全不用 LLM 当裁判，可复现。评测五个维度取无加权平均：表格结构、图表抽取、内容忠实度、格式语义、视觉定位。 https://ww

I had to look twice at this table. I know they are just numbers, but how the heck do you beat a competitor model released just a couple of days ago on some of the toughest benchmarks out there? Someth

ExtractBench is now live on @Kaggle. It tests schema-guided document extraction on the documents most likely to break downstream agents and workflows, including long record lists, noisy scans, handwri

Every document extraction system has a perception blind spot. We mapped them. For ExtractBench, we tested 14 systems on documents that weren't born digital: 1950s regulatory filings, hand-filled tax f

The Opus 5 system card itself is a fun PDF to parse. It's 193 pages and stacked with labeled and unlabeled charts 📊 LlamaParse does a surprisingly good job on agentic (1.25c per page) and agentic plus

Harrison Chase @hwchase17: Nemotron 3 Ultra hit 86%. Claude Opus hit 87%. At one-tenth the cost. Chase runs LangChain and disclosed it on stage: inside LangChain's internal deep-agents benchmark, open

Cool paper on catching alignment failures with Jev. The ideas is to ask Jev one generic yes/no question about a model's response, and use its probability as a score. With no extra training, that score

benched 5.5 is IN-SANE. It almost broke our benchmark. I thought it was a bug. Claude Opus 5.5 is the new #1 on our writing benchmark, and it is not close at all. 2631 Elo. Second one, Fable, is at 23

GPT-6 Astra is coming to Devin. On FrontierCode 1.1, Astra performs within 0.4 points of Fable 5 at a 64% lower cost. It also sets a new SOTA on our internal testing benchmark, generating more compreh

// Your LLM judge disagrees with the experts // LLM Judges can be tricky to build. Here is an interesting showcasing why: There propose a reference-full benchmark of hundreds of complete human-to-huma

Banger paper from Microsoft. It's on agent reliability in real business workflows. (bookmark it) Thinkingbox is a sandbox with isolated MCP-compatible tool sessions, plus a benchmark of 507 policy-con

Try Grok 4.6! Grok 4.7, which is a major upgrade, is coming soon. Grok 4.6 is up to 5× cheaper than GPT-5.6 Sol while delivering the exact same performance on this benchmark Both score 61 on Artificia

The most dangerous document extraction failure isn't a wrong value. It's a missing row that looks like nothing is wrong. We released ExtractBench yesterday: 370 enterprise docs, 14 systems. The hardes

Try Grok 4.6 on tough real-world tasks! imo GDPVal is probably the most important benchmark, it measures the performance of models on real world tasks Big leap in performance here to top it at a great

Quick reminder of what's ok vs not ok with harnesses used for playing ARC-AGI-3: 1. Not okay: harnesses that were custom-made to solve the benchmark or that contain knowledge about the benchmark forma

this is important and true. and if you don’t understand it you don’t really understand what’s going on. @GaryMarcus Yes — model makers quietly went from benchmarking LLMs to benchmarking LLM+harness,

ParallelKernelBench is a benchmark tool or methodology developed by Together AI for evaluating the performance of parallel kernel execution in machine learning systems. The benchmark likely measures m

If you want to stack rank LLMs/VLMs on document understanding 📄, you can through ParseBench, now live on @kaggle 📊 ParseBench is the most comprehensive document OCR benchmark over real enterprise docu

EDINET-Bench is a benchmark dataset that evaluates large language models on complex financial analysis tasks using Japanese financial statements from the EDINET database. The benchmark tests LLM capab

This was my first blog post at @llama_index! Go check out the cool diagrams and stuff Media Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a mas

muse wonder sota on hard document ocr tasks the more you know! We benchmarked Gemini 4 Argon, Muse Wonder, and Muse Spark comprehensively on hard document OCR tasks. They're all pretty good! - Muse Sp

Grok 4.7 ranks first in legal matters Today we are announcing Harvey LAB-AA v1.1 in collaboration with Harvey. This updates our scoring methodology for the Legal Agent Benchmark (LAB) to add a halluci

For their Mistral Large 4 release, @MistralAI ran a blind Surge human eval, with our expert software engineers evaluating five frontier models on coding quality. Le Chonk ranked #1 among open-weight m

one of the most challenging tasks for frontier models is being able to extract thousands of values from extremely dense tables in documents. our new Extract v2.5 agents are able to get 93%-96%+ on lon

We comprehensively evaluated 16 recent frontier VLMs - including Opus 5.5 and GPT-6 Sol/Luna - on whether higher effort led to better document parsing performance. Higher effort typically leads to imp

On September 21 2026, Yohei reported results from a hard “insect‑identification” benchmark where model accuracy spanned 23 points across the evaluated systems. The Qwen 4B + Glance model scored 0.96,

Nice work from Adobe. It's standard practice to store a fixed reference answer for every eval case. When the underlying data changes daily, that stored answer goes stale. Adobe researchers write each

Astra is one of the best models we've tested at parsing complex documents with tables, at 93.2% on ParseBench. So it seems well suited for ad-hoc analysis over table-heavy documents e.g. in finance. T

Very important paper accelerating research around AI discovery. https://x.com/omarsar0/status/2095858839534862664?s=20 New benchmark from @Apodex_AI worth digging into. TRACES scores AI systems on dis

Side note: when we released ARC-AGI-3 in March, and frontier models scored <1% on it, a few Singularitarian poasters took it as a personal insult, and got very worked up about it. They argued the benc