Search: “benchmark”

If your benchmark relies on a static dataset or sampling from a static distribution densely known at training time, then it is fundamentally measuring memorization/retrieval. Which might be fine if yo

Right alongside Cursor, Devin Desktop (Windsurf) and CLI now support GLM-5.2 as well. FrontierCode Extended is a benchmark we care deeply about for real-world engineering tasks, so it's great to see G

Claude Fable 5 thinks document parsing is beneath it It is absolutely crushing on all reasoning-intensive/long horizon benchmarks: SWE-Bench Pro, FrontierCode, GDPval, Runescape, etc. But for document

This is a super exciting release - Claude Fable 5 is the same underlying model as Mythos but with added safeguards. The benchmarks are great and it's SOTA on everything by a margin but I'll add that *

Seeing a number of benchmarks showing Opus is the best model for long-running work. Five tips for running Opus autonomously for hours/days: 1. Use auto mode for permissions, so Claude doesn’t ask for

Took me a while to figure out what all the ESMFold2 rage was about. At first, the benchmarking data didn't look super remarkable to me but it turns there are many impressive aspects: - Fully open sour

We're releasing Paris 2.0, which, to our knowledge, is the world's first decentralized trained video generation model. We benchmarked it against a monolithic model trained on the same data and compute

I've been thinking a lot about the two different groups of evals you need in general agents/agents which handle broad tasks: 1. Benchmark evals - this is a suite of up to 100 eval cases which test the

Are your benchmarks actually measuring the capability you think they measure? New paper says they probably not. Coined the 'The Evaluation Trap', it provides a vocabulary for auditing whether your eva

As much as the state of benchmarks in AI is flawed, it is so much easier to track AI progress than robotics. Not sure what you can make of all the videos of robots running races or doing laundry - are

Hello again, everyone! Our latest Qwopus3.6-35B-A3B-v1 is now live, and it is once again breathtaking! Full HF space benchmark showcase and write-up is in the comments, so you can make conclusions for

Harrison Chase expresses support for expanding AI benchmarking efforts and highlights appreciation for work being done by Harvey (likely Harvey AI, a legal AI company). The post indicates enthusiasm f

Ethan Mollick advocates for dedicated funding toward developing independent benchmarking systems for AI models, noting that evaluation is currently predominantly conducted by the labs that create the

Its getting hard to benchmark frontier agent performance on longer tasks. Repeated measurement is very expensive and there are differences between using models in harnesses versus via APIs. I suspect

Localmaxxing.com is a platform in private testing phase with plans for public release the same week this announcement was made, designed to simplify benchmark sharing by eliminating the need for users

Grok continues to lead global benchmarks: • #1 in AA Omniscience (lowest hallucination rate) • #1 in IFBench performance • #1 on BridgeBench Reasoning • #1 on BridgeBench Speed • #1 on BridgeBench Low

The chart says GLM-5.1 scored 54.9 on coding benchmarks. Three points behind Claude Opus 4.6. Interesting but not the story. The story is what trained it. Zero Nvidia GPUs. 100,000 Huawei Ascend 910B

🚀 DeepSeek-V4.1-Flash-1017 is here! Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our

Some strong embeddings models have been released recently. Ours is still better. We benchmarked Embed 5 Fast against Google's EmbeddingGemma 2 across 163 datasets using RCP-nDCG@10. Embed 5 Fast leads

Mistral AI announced in mid‑October 2026 that its new model, Mistral Large 4, achieved high scores on Ulam’s private STEM benchmark (STEMBench), particularly excelling in engineering and astronomy cat

Hermes Agent is the best place to use the new @NVIDIAAI Nemotron 3.5 Super VL and Bolt models. NVIDIA benchmarked Super VL on Hermes Bench with a score of 63.44, and compared to similar local models i

🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷 We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level software engineering. Mistral Large 4 scores 13.3%, ahead of Muse Spar

Introducing GeoGuess Bench! A benchmark to measure how good AI models are at playing GeoGuessr. We gave each model 210 photos from around the world, asked it to predict where each photo was taken, and

Introducing Hermes Index: an average of benchmarks measuring model performance in Hermes Agent. Our goal with Hermes Index is to better inform users choosing between models and to show labs how they p

Thariq (@trq212) posted on September 25 , 2026 that readers can view interactive explainers of benchmarks and demos he built on the new Claude dev site (https://claude.dev/blog). The tweet includes a

Most mental health benchmarks focus on emergency situations. MentalHealthBench is designed to cover the full spectrum of mental health conversations that people bring to AI - from everyday support to

Thanks for the model, we were able to port Laya to coreml with 99.5% of the ops on ANE + benchmarked too. it is now blazing fast with 3.7 ms per decision on an M5 Pro. Release: https://github.com/Flui

the independence of AI evaluators is going to matter more... so i made it a 'benchmark': http://evaluatorbench.com (research preview) source-linked directory of third-party evaluators of frontier AI.

ELON MUSK: I saw some pretty funny jokes on 𝕏. One of them I saw was: 'Your girlfriend's a 10, but she's a benchmark maxxer.' 🤣 Media All-In Summit: Elon Musk & Gwynne Shotwell -- AI Risks and Peer Re

deepseek v4.1 flash has arrived on together ai it beats gpt-5.6 sol on agentic benchmarks at one third the cost per task the model supports a 1m context window and native multimodal input, with 552b t

Fugu Ultra v2 by the numbers. Full Results: https://sakana.ai/fugu-max-release/ • #1 on 5 out of 8 hard benchmarks (including DeepSWE, Chartography, Toolathon) • DeepSWE: 74.3 • Chartography: 48.3 vs

Another goated release by DeepSeek. Open weight, btw. Outperforms Opus 5 on key benchmarks, btw. Lower prices too. Looks like an extremely efficient model. What a legendary run. 🚀 Introducing DeepSeek

A language model walks into a benchmark. Scores 99.9% on MMLU. Wins a gold medal at the IMO. Solves SWE-bench. Passes Humanity’s Last Exam. Resists every jailbreak from Pliny the Liberator. The first

We benchmarked DeepSeek V4.1 Flash by @deepseek_ai . It reached 98% of GPT-6 Astra’s score at 1.4% of the cost on everyday design tasks based on user requests. Every model except Astra scored lower AN

Another informal nerd benchmark that AIs had struggled with has been passed: Astra successfully designed a Magic the Gathering deck and then used it to defeat a bot on Arena. (the deck may not be stun

@GaryMarcus was right. It was the harness, not the model. That’s a much bigger deal (and hidden danger) than the benchmark score. If the real capability lives in opaque, vendor-controlled scaffolding

Other investors in the round include Benchmark, Bessemer, Kleiner Perkins, Greylock, Lightspeed, Altimeter, Bond Capital, Meritech, Atreides, Valor, T. Rowe Price, Lux, 8VC, D1, 137 Ventures, DST, Pos

This model represents a step-function improvement on many benchmarks, and its training is ongoing. Our internal model group arrived at the Navier–Stokes solution in 88 hours, using around 10,000 coord

We are excited to bring the first open benchmarking of Google's TPUs to the world Running every day, on many models + scenarios $/token is better than B200 and B300 Huge shout-out to Google @inferact

OpenAI’s new language‑model system **Astra** has achieved state‑of‑the‑art performance on three key workflow benchmarks: the Agents’ Last Exam, AutomationBench, and ScreenSpot Pro. These suites evalua

Meet E-Commerce Bench, a new benchmark for long-horizon autonomous business operations. 🚀 Agents start with ¥100,000 to run online stores for 365 days, handling sourcing, negotiation, pricing, promoti

'One remaining area of important capability we haven't seen demonstrated yet by any model is open-ended invention and discovery.' Benchmarks keep going up. Open-endedness keeps moving up in importance

Paper Instruments benchmarked sandbox providers for their RL rollouts, and E2B's tool execution came in up to 3x faster than every other sandbox provider they tested, reducing idle GPU time and traini

We benchmark Qwen3.6-35B-A3B on an M5 Max MacBook Pro. Across ten prompt lengths and ten decode contexts, Lily was faster than MLX-LM, averaging 1.23× higher prefill throughput and 1.35× higher decode

DeepSeek V4 Flash now comes with native vision on Fireworks. Available now on serverless and priority tiers. Better benchmarks than the text-only 0731 version at the same low price: 0.22 input / 0.66

Introducing Fleet: GPU benchmarking entirely in your browser. Run WebGPU compute kernels drawn from real AI workloads directly on your hardware and earn a personalized card built for your device. Rate