Search: “benchmark”

With only 6B active parameters, Qwen3.8-Flash-Next-Base tops 8 of 14 benchmarks, including MMLU-Pro, SuperGPQA, BBH and GSM8K. And it remains competitive with Qwen3.7-Plus-Base on the rest. Its 51B N-

Advances in models, chips, or devices will continually improve local-first capabilities. We will open our benchmarks soon. Read the full post: https://www.perplexity.ai/hub/blog/a-local-first-agent-fo

A year ago today we published The Last Economy by @EMostaque. Since then, scores on the hardest AI benchmarks have risen sharply, while the cost of useful intelligence has kept falling. The book asks

The real benchmark for an AI agent isn’t what it says. It’s what it gets done. AutoClaw, @Zai_org 's AI Agent for Work, is built to turn intent into finished output. Give it a complex task. AutoClaw b

#1 open-weight model on Harvey's Legal Agent benchmark! 🙌 Strong enough to handle professional tasks. Small enough to run on your local machine. Qwen3.8-27B is becoming part of your everyday workflows

It's felt like harness month on Twitter. We're seeing much faster and cheaper gains on a bunch of benchmarks by focusing on harness improvement rather than model improvement. Allowing the harness to b

Brex’s summer benchmark shows how quickly the infrastructure behind the agent economy is growing. 14 of the 25 fastest-growing software vendors serve teams building AI products, and Together AI ranks

I am bullish on simple agent harnesses. Love this idea from Vercel. fx is a new coding agent harness for model benchmarking, sandboxing, evals, and gyms. Introducing fx, a tiny, open, native coding ag

We’re excited to announce DiG-bench, a new benchmark for discovery! Over the last few weeks we’ve been testing frontier AI models on our novel discovery games and seeing how they score. Each game is a

Skill libraries are shipping in agent harnesses on the assumption that writing skills down compounds. A new benchmark tests that directly. ContinualSkillBench covers five domains, each with 100 interc

Nice benchmark to measure agentic e-commerce capabilities. They ran an agent for one simulated year of e-commerce operations and it ends up with 27.3% of the money a human makes. MerchantBench is a 36

🚀 DeepSeek-V4-Flash Official API is now LIVE in public beta! 🔷 We’ve massively upgraded its Agent capabilities—benchmark scores are now far surpassing the V4-Pro-Preview. Check out the massive perform

A benchmark score reflects the model as well as the harness and settings used to run it. For long-running agents, retaining reasoning and compacting context lets the model build on what it has already

BREAKING: Grok 4.5 (high) ranks #1 on the HighWalk benchmark, which tests how well AI agents update technical specifications from code changes. Grok delivered the best combination of quality and opera

BREAKING: Grok 4.5 just claimed the top spot on the new HighWalk Benchmark. The independent test measures how well AI models update real technical specifications from 46 Laravel commits — heavy on cod

GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what

On FrontierCode 1.1 Extended, our benchmark for real-world engineering tasks that grades mergeability and quality, Kimi K3 scores 58.2% with a 63.6% pass rate. Within Devin, it excels on reproducing b

One of the least explored questions in AI is how to build rich open-ended environments for learning. Benchmarks evaluate intelligence, environments shape it. iLands is interesting because it doesn't j

Really interesting concept -- instead of optimizing against static benchmarks or internally generated reward models, agents are evaluated through real economic interactions. Using an external market a

training to benchmarks ≠ getting to AGI This suggests that Opus 5 's gain on ARC-AGI-3 was the result of specific training to improve on that eval, and not a generalized increase in abstract reasoning

An interesting thing I'm observing from the blog/system card is that on a good chunk of the reported benchmarks (~20-30% from a skim), Opus 5 max thinking leads to a degradation in performance compare

We benchmarked Gemini 3.6 Flash and Gemini 3.5 Flash Lite on document understanding. We compared against their prior versions - Gemini 3.5 Flash and Gemini 3.1 Flash Lite. 1️⃣ Gemini 3.6 Flash has rou

Interesting surprise drop from Thinky! The Inkling model looks pretty solid on benchmarks, and it has some little surprises in its architecture: - Small conv layers in several places - An RMSNorm for

The best base model for training isn't about someone else's benchmarks. Excited to offer @thinkymachines' first open-weights model: Inkling. 975B MoE, multimodal, with Apache 2.0. A great foundation f

Thinky with a ~1T param, 41B active, apache-2 model Benchmarks are a clear step up from Nemotron Ultra (55B active), new best American model, and omni input. A bit behind GLM 5.2 on agentic benchies,

Trending AND fastest-growing in the same month? Benchmarks change daily, but only @tryramp has the database of real receipts to track this. We can confirm: demand for open-weight inference and trainin

@OpenAI released GPT 5.6 today and we ran a day 0 benchmark in ParseBench to test improvements in document understanding. The new family of models continues to excel at reading text and tables, but co

Not another demo or benchmark. @ShoucongChen is a senior member of our technical staff. A real project, scoped at 1-month. Delivered in 4 days with GLM5.2 Fast. The best devs deserve >400 t/sec. Take

We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability. We find 30% of SWE-Bench Pro tasks to be broken, and ar

We tuned the harness for @NVIDIAAI Nemotron 3 Ultra. Benchmark-leading performance. 10x lower inference costs. ✅ An aggregate score of 0.86 at a cost of 4.48 ✅ The closest-performing model: 43.48 http

MAI-1 has no independent benchmarks yet, but the ones they released suggest it is worse than Sonnet 4.6. Not sure it is going to be great as a Copilot at Excel and Outlook, especially as there are alr

“This study cuts through the optimism surrounding medical AI by showing how easily benchmark success can be mistaken for real readiness. In medical AI, impressive scores are clearly not the same as tr

With our internal coding benchmark, we're able to confidently introduce open-weight models into our AI code reviewer w/o degrading code quality. Have the frontier model (Fable) to the hardest work, de

Today, we are releasing Le Chaton L∃∀N, aka Leanstral 1.5. It achieves SOTA performance on graduate algebra benchmarks FATE-H and FATE-X and improves Pareto Frontier on PutnamBench, solving 587/672 pr

harbor is a great framework for running evals for long running, stateful agents its becoming industry standard, powering benchmarks like terminal bench 2 we've integrated deeply with harbor: across La

On FrontierCode (Extended), our benchmark for real-world engineering tasks that grades mergeability and quality, Sonnet 5 scores 53.8% and has a 57.6% pass rate (higher than Opus 4.8). Note: These rel

We’re introducing GeneBench-Pro, a research-level benchmark for a harder kind of AI progress: how well agents can navigate messy biological data, choose the right analysis path, and make judgment call

Conventional model routing sucks. It passes benchmarks but fails to write code you'd actually merge. Introducing Devin Fusion, a new hybrid-model harness for agentic coding. In testing, it reduces the

This post by Filip Hric explains how to interpret AI code review benchmarks, specifically breaking down key evaluation metrics like precision and recall that are commonly used to assess code review to

We're sharing new research on how models hack public benchmarks. The latest models, including Opus 4.8 and Composer 2.5, learn to retrieve solutions from the internet or git history. When we apply a s

We benchmarked Mistral OCR against other frontier and open-weight models on ParseBench 📊 For a model at its price point, it is quite competitive! - It wins on semantic formatting - understanding strik

Benchmarks tell only part of the story. Fugu’s real value shows up in long, messy, real-world workflows. During our beta with 500 users, we saw Fugu Ultra drive meaningful progress in fully automated

GLM-5.2 leads open weights models and sits at #3 overall on GDPval-AA, a real-world agentic work benchmark GLM-5.2 from @Zai_org scores 1524 Elo on GDPval-AA, which measures performance on real-world,

Pamela Fox created a custom t-shirt design referencing Simon Willison's Pelican benchmark, a technical tool or metric in web development or performance testing. She posted about it on X (formerly Twit

We've kept hearing how GLM-5.2 beats Opus 4.8, and are skeptical of benchmarks - so we tested them on a real bug from the Cline repo. While both models fixed the issue, GLM was the winner in terms of

Cohere has released Transcribe, an open-source speech recognition model that achieved the top ranking on Hugging Face's newly established Far-Field Automatic Speech Recognition (ASR) benchmark. The mo