Search: “benchmark”

Introducing Muse Glimmer, an open-weight 30B-parameter model optimized for local, always-on agent workflows. Muse Glimmer delivers strong performance on key agentic use cases and benchmarks compared w

The best 'raw' frontier model for document parsing is gemini 3 flash, but the issue is that since then the flash models have gotten 3x more expensive while flatlining on visual recognition across comp

Researchers from TogetherAI analyzed DeepSeek V4 Flash and GPT‑5.6 Luna on the DeepSWE benchmark. The study found that employing a DeepSeek‑first cascade with test‑suite verification solved more tasks

Codex is overoptimised for large models: it ranks 2nd out of 10 for GLM 5.2 but drops to 9th place for Gemma-4! Almost all the effort in this field goes into tuning the weights. We wanted to know how

Document OCR is not Getting Commoditized (by Frontier Models) The most common question I get is whether frontier models are going to eat all document processing solutions - just screenshot the page an

Qwen3.8-Max from Alibaba’s Qwen team achieved second place in the Vision Arena benchmark, scoring 1,305 points. It trails only Claude Fable 5 (High), which leads by a slim 13‑point margin. The post un

.@ssankar says Palantir was able to make Nvidia's Nemotron Ultra model 'better than frontier': 'I literally almost felt gaslit when, within 24 hours of getting Nemotron up with no post-training, this

Try Grok 4.5 http://X.ai/cli BREAKING: Grok 4.5 outperforms GPT-5.6 Terra across almost all shared benchmarks on AskClash, leading in ACB, GPQA, SWE-P, and Atlas while also achieving a higher overall

And Grok 4.6 comes out in a week BREAKING: Grok 4.5 ranked #1 on LaurenBench with a score of 56.9%, ahead of Claude Sonnet 5, GLM 5.2, Claude Opus 5, Kimi K3 and GPT-5.6. The benchmark tests real-worl

BREAKING: Grok 4.5 ranked #1 on LaurenBench with a score of 56.9%, ahead of Claude Sonnet 5, GLM 5.2, Claude Opus 5, Kimi K3 and GPT-5.6. The benchmark tests real-world AI agents across conversation,

SpaceXAI has released its new Grok Voice Think Fast 2.0, which on the Artificial Analysis Speech‑to‑Speech benchmark outperformed leading models from OpenAI, Google, Alibaba and DeepSlate. The claim w

Announcing Fugu-Ultra v1.1 🐡 We’ve been thrilled by the reception to the Fugu model family. Thanks to everyone who tried it, shared feedback, and trusted Fugu with real work. Today, we’re releasing Fu

Many still debate open vs closed, and compare cost per token (accounting metric). Better to shift attention to cost per successful task. @seldo and the team at @arizeai did so across 2,400 runs. Concl

Holy shit wow We're partnering with @huggingface to investigate an unprecedented security incident. Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation. Shari

I wrote about the completely wild incident where OpenAI were testing a new model and it broke out of its sandbox and broke INTO Hugging Face to steal the answers to the benchmark https://simonwillison

We're partnering with @huggingface to investigate an unprecedented security incident. Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation. Sharing preliminary

We trained and released DSpark speculators for Kimi-K2.6 and Kimi-K2.7-Code on @huggingface, with native serving support in @vllm_project. Across six benchmarks in our batch-size-1 evaluation: Kimi-K2

Grok places second after Fable on real-world software engineering Grok 4.5 from @SpaceXAI places #2 on the APEX-SWE leaderboard at 51.2% Pass@1 (±6.0), behind Fable 5 (65.5% ±6.2) on our benchmark for

Grok Build Grok 4.5 with Grok Build just ranked #1 on the SWE-Atlas-QnA benchmark with a score of 84 That puts it level with GPT-5.6 (max) Codex and ahead of Claude Code Fable 5 (max), Opus 4.8 (max),

Grok is closing the loop on real-world use cases @OpenAI released 3 new models yesterday and we immediately tested it on our internal benchmark. All three outperforming gpt-5.5 but @SpaceXAI still the

Grok 4.5, xAI's large language model, has achieved the top ranking in the SWE (Software Engineering) Marathon benchmark, according to an announcement by Elon Musk. This ranking suggests the model demo

Grok 4.5 is dominating the latest AI leaderboards Claims the #1 spot: • #1 on AutomationBench-AA • #1 on Terminal-Bench v2 • #1 on Harvey Legal Agent Benchmark • #1 on SWE Marathon • #1 on SWE-Atlas-Q

Cursor shared a link to their evals page that provides a comprehensive comparison of different AI models' performance and capabilities. The page likely displays benchmarks, metrics, or evaluation resu

Watch Ollama's @jmorgan and @peterfenton on @CNBC with @dee_bosa at 12pm PT / 3pm ET to discuss 'The Post-Frontier Era' and 'Open-Source AI's Breakout Moment' Benchmark’s @peterfenton says 90%+ of tok

Introducing the NemoClaw Deep Agents Blueprint, a reference architecture for building open agent systems developed with @NVIDIA ✅ A fully open stack enterprises can own and customize ✅ Benchmark-leadi

Another day another big public tech co saying they’re using open source AI models extensively With our internal coding benchmark, we're able to confidently introduce open-weight models into our AI cod

GLM 5.2, running on Together AI's platform, has achieved the top ranking on Artificial Analysis benchmarks for both output speed and latency metrics. This announcement highlights Together AI's infrast

Francois Chollet discusses how careful observation of ARC-AGI benchmark performance can enable early predictions about AI system capabilities and progress. The post likely highlights patterns or trend

François Chollet observed that all leading contenders on the ARC-AGI-3 benchmark employ a similar methodological approach, suggesting a convergence in techniques for solving abstract reasoning tasks.

DSGym is a comprehensive framework designed to evaluate and train AI agents for data science tasks, providing a structured environment for benchmarking agent performance across various data science wo

Multi-GPU kernels are the real test for coding models. Today at @aiDotEngineer, @simran_s_arora shared ParallelKernelBench, an open-source benchmark for evaluating whether LLMs can write fast CUDA ker

Fireworks AI announced achieving 446 tokens per second throughput speeds as measured by Artificial Analysis benchmarks, positioning this as their standard performance metric. The company offers custom

The harness can be adapted for other configurations & can also achieve sota performance at lower cost https://ii.inc/blog/post/zenith-research FrontierSWE is probably the hardest practical benchmark o

It is worth being very, very careful about how you are approaching routing, especially when the systems are primarily tested on verifiable IT benchmarks, which may lead you to overestimate the ability

The more serious answer @emollick I actually wouldn't be surprised if we're generally in an AI model release moratorium (regardless of capabilities) until the cybersecurity benchmark is finalized. Not

Nous Research announced an upcoming HermesBench full leaderboard, indicating they are developing or expanding a benchmarking system, likely for evaluating their Hermes model family or related language

Nous Research is conducting benchmarking tests to evaluate whether combinations of open-source models can achieve performance comparable to Anthropic's Claude Opus while maintaining significantly lowe

In this episode, @OpenAI Chief Research Officer @markchen90 joins @allenpark to flambé shrimp, cook Korean stew, and chat about being at the frontier of AI research: why scaling laws and pre-training

LLMs are getting better at writing GPU kernels. Multi-GPU kernels are the harder test. At @aiDotEngineer World's Fair, @simran_s_arora will share ParallelKernelBench, an open-source benchmark built fr

Cursor discusses methods for improving AI evaluation environments to prevent reward hacking and ensure benchmark scores more accurately measure model capabilities rather than exploitation of evaluatio

GLM-5.2 achieved token generation speeds of 318 tokens per second, surpassing the previously reported 280 tokens per second benchmark, with indications that further performance improvements are possib

5.2 could be better with more RL ... Deepswe's benchmark results are my own experience. I've used all models, GLM 5.2 ≈ Claude Opus 4.6–4.7. Kimi 2.7 code more like inference optimization. Looking for

Try Kimi K2.7 and GLM 5.2 for free in Devin Desktop and CLI Kimi K2.7 Code and GLM 5.2 are available in Devin Desktop and CLI Both perform strongly on FrontierCode Extended, our benchmark for real-wor

LLMs write fast single-GPU kernels. Ask for a multi-GPU one and they fall apart. ParallelKernelBench (PKB) measures how they fail by benchmarking against 87 problems pulled from real codebases includi

Fugu stands shoulder-to-shoulder with leading models like Fable and Mythos across the industry's most rigorous engineering, scientific, and reasoning benchmarks. Read the full blog: https://sakana.ai/

Today on AI:AM: @swyx, who coined 'AI Engineer', is gearing up for his next @aidotengineer World's Fair next week We'll be talking about his new FrontierCode benchmark, the state of SWE, whether AI co