Search: “benchmark”

I trust these vibe checks more than I trust most of the benchmark results GPT-6 astra seems more powerful for e2e automation and better at writing (a massively underrated gap for these models). But it

GLM-5.3-Flash is live on Fireworks on day… 2 Why? Because we take quality very seriously. We found a benchmark discrepancy we couldn’t explain, so we delayed the launch to investigate. Day 0 (Wed): we

We benchmarked 20+ open-weight models on easy-to-hard document extraction tasks through ExtractBench. The results are all available on @huggingface 🤗 ExtractBench is a schema-guided extraction benchma

On August 7, 2026 Ethan Mollick tweeted that “every remaining good AI benchmark score has an implied asterisk next to it which reads: * could be significantly higher with a better harness.” The commen

We benchmarked Opus 5 comprehensively on document understanding through ParseBench. It is roughly on par with Opus 4.8 - it does a few points worse on dense tables, but does slightly better on parsing

Introducing AI IQ Bio: the most comprehensive set of biotech benchmarks in the world ...& Bio IQ: the most comprehensive 'biotech capabilities index' ever produced Benchmark sources include benchmarks

There are a lot of coding and reasoning benchmarks for AI agents, but not a lot for document understanding - which is a prerequisite for all downstream knowledge work. We released ParseBench ~a month

I find that open weights models over-perform on benchmarks compared to actual real-world usage, and Kimi feels like no exception. For example, a small amount of use will show that Kimi is not as good

Your users say search is bad. How do you turn that complaint into an eval? Aaron Linsky explains how Bridgewater built a benchmark of about 200 examples in under a week, then used it to measure search

Choosing the right AI model is an operating decision. Today, we’re sharing our first set of benchmarks drawn from work across the 75+ accounting and IT services firms that make up Thrive Holdings. We

Recommended benchmark. I expect voice to become one of the main ways people interact with robots and physical AI. That makes speech recognition in real conversations much more important than it seems

Claude Opus 5.5 is now available as the Standard effort level in Perplexity Computer. On our WANDR benchmark, it scored 0.610 at $4.13 per task, slightly outperforming Claude Fable 5.1 while costing 6

We're introducing Q2D-Web (Query2Doc-Web), a benchmark and public leaderboard for evaluating retrieval in agentic RAG systems. Q2D-Web tests how embedding models perform on large-scale web search usin

Our general-purpose coding agent just scored 100% on the ARC-AGI-3 interactive reasoning benchmark. NVIDIA AVO completed all 183 levels across all 25 public environments, figuring out what to do with

When you move a model into production, you want the quality you evaluated to carry through the serving stack. @Kimi_Moonshot benchmarked Kimi K3 across major inference providers, and Together AI ranke

Also I think AISI is a great model of a government agency tasked with AI security. They have open benchmarks, very fast testing, and clear communication about incidents that is neither hyped up nor hi

Ethan Mollick highlights that large, currently under‑explored benchmarks (e.g., ARC‑AGI, GDPval, METR long horizons, ASI cyber tasks) are growing in complexity, yet they increasingly lack systematic h

On FrontierCode 1.1, our benchmark for real-world engineering tasks that grades mergeability and quality, Opus 5 scores 63.6% with a 69.6% pass rate on Extended — approaching Fable 5 at half the cost.

GROK 4.5 LEADS ON REAL PROFESSIONAL WORK BENCHMARK New data from Snorkel shows Grok 4.5 outperforming other frontier models on real-world professional tasks. On their GDPval+ benchmark (expert-created

The metrics discussion at OpenAI is a little confusing to me. I appreciate the clarification about bad benchmarks, but they spent a lot of money developing a very good benchmark of autonomous model ab

For anyone who isn't sure, this is how you release a model and talk about the performance. Not 3-5 cherry-picked benchmarks. Performance：Qwen3.7-Max performs strongly across benchmarks in coding agent

DeepSeek released a new open-weights model with strong benchmark performance, though Mollick notes that benchmark results may not fully capture the capabilities of open models compared to closed syste

Ocr benchmark We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that works with real-world files. ParseBench is

The entire team put a lot of effort into this benchmark. Document parsing and OCR certainly isn't solved, but now it is a bit easier to measure 🦙 We’re open sourcing the first document OCR benchmark f

We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that works with real-world files. ParseBench is a benchmark t

Benchmarks are becoming the core competency of any company. Our company mission is to build the world’s best models for document processing. This also means that we have to build the world’s best benc

Zai recently launched a domain‑specific language model called **GLM Legal**. According to a tweet posted at 4:13 PM on Oct 12, 2026, GLM Legal became the leading open‑weight model on both **Harvey’s L

The ultimate LLM benchmark: Chilean Spanish. 80% of AI models fail to recognize Chilean Spanish and insist on speaking like an Argentine 80% de modelos de IA no logra reconocer el español chileno e in

Prinz has been running legal benchmarks against AI systems and the latest models are very good... but cheaper/low reasoning models are not very good. So, how do you make sure your vendor selling an AI

Grok 4.6 takes top spot on this benchmark 🥇 We evaluated Grok 4.6 on MedAgentBench, a benchmark for agentic clinical EHR tasks, and it took the top spot. Grok 4.6 posts the highest pass@1 we've measur

Grok 4.6 earned first place on RuntimeWire’s **Newsroom Reliability v0.2** benchmark, scoring 0.79 and surpassing competitors such as GPT‑5.6 Sol, Claude Opus 4.8, Gemini, and DeepSeek. The benchmark

Grok 4.6 just matched Claude Fable 5 on Perplexity’s WANDR benchmark....at more than 60% lower cost Both score exactly 0.496 But the cost per task is ridiculous: • Grok 4.6 → 7.58 • Claude Fable 5 → 2

SPACEXAI: Grok 4.6 leads on the two strongest knowledge-work / real-world productivity benchmarks (GDPVal-AA and AA-Briefcase) and on the legal benchmark, while remaining highly competitive on coding-

The model punches above its weight, outperforming Gemma 4 E2B and Ministral 3 3B across a broad range of visual understanding benchmarks and delivers particularly strong results on document understand

I agree with what this AI paper suggests. Self-improving agents should evolve their benchmarks too. (bookmark it) Self-improving agents are one of the most important directions in AI right now, and mo

As AI coding models advance in capability, current evaluation benchmarks must evolve to remain challenging and meaningful measures of progress. OpenAI argues that improved benchmarks need to be harder

Reporting benchmark results as a single scalar percentage is insufficient for meaningful evaluation of model performance. Comprehensive benchmark reporting should include efficiency metrics alongside

ParallelKernelBench is a benchmarking framework designed to evaluate large language models' ability to generate optimized GPU kernels for multi-GPU computing environments. The benchmark assesses LLMs

You really need to benchmark models for your use case. As soon as judgements & decisions stack on top of each other, the differences between models amplifies, and no standard benchmark will tell you t

We're presenting ParseBench at CVPR 2026! ParseBench is the most comprehensive document understanding benchmark for VLMs. ✅ It contains 2k pages of real-world enterprise documents ✅ It has comprehensi

WANDR is our in-house wide benchmark, built to mirror real professional research workloads. Search as Code scores 0.386 to the next best system's 0.152, and the benchmark is far from saturated. We're

I think Epoch does a great job benchmarking, but I continue to believe that open weights models are much more fragile, especially out-of-distribution, than their benchmarks indicate. Vibe-wise, I don’

ran my first benchmark this weekend (longmemeval) mostly to test activegraph, learned a lot! - this is a stepping stone to show the event based agent system works. the AI convinced me not to start wit

Most models are only evaluated on a fraction of the benchmarks out there. ArtifactLinker, our new system, predicts which ones would set a new state-of-the-art on benchmarks hosted on @HuggingFace, the

How do you know your document parser is ready for production? 🤔 Existing benchmarks miss what AI agents actually need. That's the gap ParseBench, the first doc OCR benchmark for AI agents, fills. We'l

Kimi K2.6 is now open-weight #1 on Finance Agent Benchmark V2. Can AI do the job of a financial analyst? We just released V2 of our Finance Agent Benchmark and tested the frontier models. The results