Search: “benchmark”

💫Very happy to release NeuralBench, to benchmark Neuro AI models and datasets in the open! 🧵Thread, 💻Code, 📝White Paper below: 🧠 Introducing NeuralBench: a unified, open-source framework to benchmark

GPT-5.5 is likely the best model in the world. But open models like Kimi and Minimax get almost identical coding benchmark scores at 10-25x lower cost. I broke down the benchmarks and pricing. Here's

ParseBench is now live on @Kaggle. The first document OCR benchmark built for AI agents — 2,000 enterprise pages, 167K+ test rules, 5 dimensions that actually break downstream agents. Benchmark your p

Agent evals are drifting away from production reality. Most benchmarks use clean tasks, well-specified requirements, deterministic metrics, and retrospective curation. Production work is messier, with

The power of OpenDocRouter is already working: Anthropic just dropped Haiku 5.5 and we already have initial benchmark results. I see pretty comparable results to Luna overal. Slightly better at tables

we comprehensively benchmarked gpt-6.1 sol on document OCR tasks. compared to gpt-6 sol from a week ago (!), we see a sizable increase in table parsing capabilities and reading order, which is pleasan

We let open models access the outside world during evals on tasks from public benchmarks. Here's what happened: Most went and found upstream commits containing fixes for the tasks they were supposed t

Model personality matters in a way that benchmarks can't capture. Opus models went through a rough patch from around 4.7 to 5 where they just didn't feel 'Claude-y' anymore, more like an watered-down

Some food for thought when designing benchmarks... So, here's a little computer-use (visual) comparison between GPT-5.6 Astra and Qwen3.8 Max. The task here was to recreate the image in the center usi

Really strong benchmark paper on coding agents. Claude Opus 5 running under Claude Code passes 23.9% of the evaluations. An expert human reference scores 82.2%. Here is what the task actually is. The

Perplexity AI has released its new language model GLM 5.3 on Perplexity Computer. Designed for long‑context, multimodal agent workloads, GLM 5.3 outperformed the previous GLM 5.2 on the company’s WAND

Great paper on why agent leaderboard comparisons are hard to trust. It's on the hot topic of how much of an agent benchmark score actually belongs to the harness. The harness is the layer between the

We built one of the most comprehensive benchmarks for document extraction, and evaluated it across a lot of different systems: ✅ one-shot frontier VLMs ✅ frontier VLMs + coding agent harnesses ✅ one-s

Social science has long addressed the challenge of evaluating qualities that cannot be objectively verified—such as writing quality or creative ideas—and its methodologies remain relevant for today’s

2/ a cost benchmark showed the same coding task running three to four times cheaper, depending purely on the harness wrapped around the model. Same intelligence, wildly different accuracy and cost, de

“Pelican on a bicycle” LLM benchmark by @simonw 2024 (https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/) is now “Call of Duty” 2026… Claude Opus 5 one-shotted this game. EVERYTHING you see

Am I supposed to interpret that it's better than Fable 5 from the benchmarks? Or it's ~close but cheaper? Introducing Claude Opus 5. It's a thoughtful and proactive model that comes close to the front

BREAKING: Grok 4.5 has climbed to #2 on the FrontierSWE benchmark. The result places Grok 4.5 among the world's top-performing AI models for software engineering tasks, highlighting its growing streng

You really need your own benchmarks. If you are translating hieroglyphics, use Gemini 3.5 Flash. If you are running a vending machine use Opus 4.8. (This is one reason why I am skeptical of just swapp

📢Qwen3.7-Max just hit #3 on ITbench-AA — a fresh benchmark testing how well models handle real-world enterprise IT tasks, agentic-style. 🔧Agentic era, go with Qwen.🏃🏃 Artificial Analysis and IBM Resea

'One thing that we've been seeing recently is that inference benchmarks don't really match production workloads that well.' - @realDanFu, VP of Kernels When you're running dozens of concurrent coding

HF leaderboards are fantastic, thank you @ClementDelangue @Thom_Wolf @NielsRogge for making a critical facet of benchmarks so easy! Check out http://swebench.com as well The SWE-bench Verified leaderb

Grok achieved top performance on the τ-voice Bench voice AI benchmark, scoring 67.3% overall and leading in specific domains including retail (62.3%) and airline (66%) categories. The announcement was

benchmarking agents by their ability to play video games 🎮 We are excited to launch VideoGameBench on Antim Labs, created by @a1zhang, Thomas L. Griffiths (@cocosci_lab), @karthik_r_n, and @OfirPress

Who is running local models on GPUs on OpenClaw? I have started benchmarking different models this week. I am working on improving model selection and switching UX on OpenClaw, i.e. I run /model vllm/

ParseBench is the most comprehensive OCR benchmark for real-world enterprise documents: financial filings, contracts, insurance documents, and more. We evaluate across 5 dimensions that are present am

The new model from Meta is already looking like a disappointment: overoptimized for public benchmark numbers at the detriment of everything else. Knowing how to evaluate models in a way that correlate

A lot of you are asking about the :cloud tag. Here's the deal: GLM-5.1 is a 744B parameter beast. To hit that 54.9 benchmark score without turning my M4 Mac Mini into a space heater, I run the cloud e

This naming scheme is bonkers (minor model numbers and date tags) and the benchmark step is big. Could prove stronger than GLM 5.3 with vision built in. The King of open weights is back? 🚀 DeepSeek-V4

We benchmarked Gemini 4 Argon, Muse Wonder, and Muse Spark comprehensively on hard document OCR tasks. They're all pretty good! - Muse Spark and Gemini 4 Argon* are SOTA at their respective price poin

OpenDocRouter is here! https://www.opendocrouter.ai/ Some time ago, I realized LlamaIndex was really good at benchmarking and applying nearly every model to document parsing (see: ParseBench). So, why

decision models + local ai = ❤️ tried Laya (421M) on llama.cpp in a little apple-picking benchmark, it's almost as good as Qwen 4B at 12x the speed. Media Decision models in llama.cpp are now availabl

In September 2026, Sebastian Thrun announced **PhilosophyBench**, the first independent, large‑scale benchmark created by Stanford’s AI Lab and HCI group to evaluate artificial intelligence’s philosop

Today, @FactoryAI joins the Specialized Intelligence Index with Legacy-Bench! The SII keeps growing, now with benchmarks testing how models handle the legacy software behind payroll, financial settlem

MiMo-V2.6 is 'simply' the best (for now). Despite its simple architecture design it's currently No.1 in the open-weight benchmarks (weighted average). With 'simple,' I mean a classic Grouped Query Att

Grok 4.7 xHigh is now at the top with just ONE point away from Claude Fable 5.1 Max on Artificial Analysis’ AA-Briefcase benchmark Claude Fable 5.1 Max — 59% Grok 4.7 xHigh — 58% Just a 1-point differ

Live now: our Inference Engineering Track from AI Engineer World's Fair 2026. A benchmark tool told to run 200 queries a second that ran 38 and reported 200. A model that answered one prompt in a thou

Which AI models are most likely to cheat when given the chance? To find out, we built CheatBench [https://cheatbench.ai] : a benchmark that tests whether agents attempt to cheat when given difficult t

Just because someone puts the word AGI in their benchmark doesn't make it even remotely a reflection of what the general public understands by this term. As a layman, I define AGI as a virtual charact

imo GDPVal is probably the most important benchmark, it measures the performance of models on real world tasks Big leap in performance here to top it at a great price, congrats to @SpaceXAI team & loo

My conversation with @ericvishria of Benchmark. Eric has spent a decade investing across software and hardware, backing companies like Fireworks, Sierra, Sunday Robotics, and Cerebras. This one is abo

One of the things the pelican benchmark is still useful for is visually representing (to a tiny extent) the improvements in a single model family Here's Meta AI's Spark (8th April), Spark 1.1 (9th Jul

We are releasing PerceptionBench, a benchmark that isolates visual perception and evaluates it as a set of atomic capabilities - discovered from how today's models fail, rather than defined in advance

Kimi K3 is second only to Fable 5 on AA-Briefcase, our agentic knowledge work benchmark, but costs more than Opus 4.8 to run while averaging nearly an hour per task Last week @Kimi_Moonshot released K

**IssueBench** is an internal benchmark created by LangSmith to assess the performance of *Engine*, a continual‑learning agent that scans other agents’ traces to identify, cluster, and fix issues. In

Grok 4.5 achieved the highest score on the SWE‑Atlas‑QnA benchmark, surpassing Claude Fable 5 and GPT‑5.6 Sol, according to a Polymarket tweet posted on 13 July 2026. The update highlights Grok 4.5’s