Search: “benchmark”

People on X: “CLAUDE IS BACK, baby!” Listen, Opus 5.5 is incredible and probably my favorite overall model right now, but it is still ridiculously verbose and over-justifies and over-qualifies everyth

And the incidents apparently continue. It is worth noting how much of this is agents trying to accomplish their goals during testing by reward hacking (which sometimes seems to include actual hacking)

OpenAI announced the launch of **dots**, an always‑on agent system powered by the GPT‑6 Astra model. According to OpenAI, dots are “remarkably capable” agents designed to handle a wide range of tasks

Grok 4.7 is #1 on the AA Cyber Index Grok 4.7 xHigh ranked #1 on Artificial Analysis’ Cyber Index Grok 4.7 is now very powerful with serious cybersecurity capabilities It outperformed Claude Opus 5.5,

AI solved open mathematical problems. But the biggest surprise came afterwards. A team at Google DeepMind has used AI to resolve 9 open Erdős problems, including questions that had remained unsolved f

GPT-4 was the undisputed best model in the world for a full year and we all debated whether anyone could catch up or if OpenAI had some secret formula that nobody else knew about. Now there are are 19

After much testing and talking to actual users across all three proactive AI agents: Muse - best one for your parents, heavy IG/FB users, people who like or need easy setups, people who don’t want to

A worthwhile (and unnerving) reminder from Anthropic that we should be paying attention to AI ability thresholds, not just steady climbs In this case, the threshold is the capability needed for a self

Another great paper from Salesforce AI Research. The finding is that a 31B open Gemma-4 web agent scores 74.6% on the 9-app WebArena Infinity set, above Gemini 3 Flash with browser use at 70.1%. They

AI solved open mathematical problems. But the biggest surprise came afterwards. A team at Google DeepMind has used AI to resolve 9 open Erdős problems, including questions that had remained unsolved f

We benchmarked Gemini 4 Argon, Muse Wonder, and Muse Spark comprehensively on hard document OCR tasks. They're all pretty good! - Muse Spark and Gemini 4 Argon* are SOTA at their respective price poin

On October 13 2026, researcher Ethan Mollick tweeted that observing Gemini 4’s behavior reveals how many other LLMs are likely distilled from Claude or GPT—and thus use a similar training pipeline—mak

GPT-4 was the undisputed best model in the world for a full year and we all debated whether anyone could catch up or if OpenAI had some secret formula that nobody else knew about. Now there are are 19

Banger paper from Google DeepMind on co-evolving an agent's harness and its weights. (bookmark it) If you already log your harness changes and results, those logs can become training data for the mode

muse wonder sota on hard document ocr tasks the more you know! We benchmarked Gemini 4 Argon, Muse Wonder, and Muse Spark comprehensively on hard document OCR tasks. They're all pretty good! - Muse Sp

Benchmarks are becoming the core competency of any company. Our company mission is to build the world’s best models for document processing. This also means that we have to build the world’s best benc

Banger paper from Google Cloud AI Research. If you follow autonomous research agents, this one is worth your time. ScientistTwo takes a problem from a human expert and runs the full discovery cycle wi

MiMo-V2.6-Pro debuts as the top open weights model on the Artificial Analysis Intelligence Index (46). At 0.13 per Intelligence Index task, it lands on the Intelligence vs. Cost per Task Pareto fronti

Must-read paper from Google on self-improving agent harnesses. If you auto-optimize your agent's harness, your eval score can go up while the agent gets worse on real tasks. This paper shows how to pr

What is a way to test human-level generality in artificial intelligence? 'The meta-benchmark of being able to pass ARC-AGI-(n+1) immediately upon release' @Jabaluck @DAcemogluMIT The meta-benchmark of

The AI research community released a set of notable papers between September 28 and October 4, highlighted in a weekly roundup. Key contributions include JAZ, CASD, Agensh, Jev‑Mem, AutoGym, Taste‑Ben

decision models + local ai = ❤️ tried Laya (421M) on llama.cpp in a little apple-picking benchmark, it's almost as good as Qwen 4B at 12x the speed. Media Decision models in llama.cpp are now availabl

The ultimate LLM benchmark: Chilean Spanish. 80% of AI models fail to recognize Chilean Spanish and insist on speaking like an Argentine 80% de modelos de IA no logra reconocer el español chileno e in

Brilliant paper on agentic continual learning from Meta and MIT. (bookmark it) Agent memory harnesses usually only write forward. A lesson goes into the bank and is never checked against whether it he

Meet Mistral Large 4, aka Le Chonk. • 1T parameters, natively multimodal. 49B active. It is the best open weights model from US or Europe on aggregated benchmarks. • State-of-the-art on critical workl

For their Mistral Large 4 release, @MistralAI ran a blind Surge human eval, with our expert software engineers evaluating five frontier models on coding quality. Le Chonk ranked #1 among open-weight m

MistralAI announced that its model **Mistral Large 4** ranked as the #1 open‑source (OSS) model on **Harvey’s Legal Agent Benchmark (LAB)**, an evaluation benchmark for legal AI agents. The tweet post

Mistral AI introduced **Mistral Large 4** in October 2026 as a general‑purpose agentic model capable of gathering information and producing final deliverables across complex workflows. The launch high

Mistral OCR is on OpenDocRouter! For its price, it is decent over tables and reading order. It's also blazing fast at p50 1.8s per page We're constantly bringing in new models since OpenDocRouter laun

🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷 We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level software engineering. Mistral Large 4 scores 13.3%, ahead of Muse Spar

Mistral AI announced its Le Chonk model, which performs visual grounding by accurately locating objects within images. On the Dense 200 benchmark it achieved a 42 % score, outperforming even frontier‑

Benchmarks are becoming the core competency of any company. Our company mission is to build the world’s best models for document processing. This also means that we have to build the world’s best benc

MiMo-V2.6-Pro is live on Wally. Day 1. @XiaomiMiMo's new open-weights flagship, top of the @ArtificialAnlys Intelligence Index for open-weight models. 320 tok/s, 2.4x the model maker's own API, in <24

Recommended benchmark. I expect voice to become one of the main ways people interact with robots and physical AI. That makes speech recognition in real conversations much more important than it seems

Elliot Arledge released a video titled “Claude Opus 5.5 just one‑shot,” which explains how to optimize CUDA kernels using a music‑video format. The clip was posted on September 27, 2026 and has accumu

BREAKING: Grok is helping power America .gov, the U.S. government’s new AI website. 🇺🇸 U.S. Chief Design Officer Joe Gebbia says Grok & Gemini power the site. Americans can ask questions and get answe

Grok 4.7 is #1 on the AA Cyber Index Grok 4.7 xHigh ranked #1 on Artificial Analysis’ Cyber Index Grok 4.7 is now very powerful with serious cybersecurity capabilities It outperformed Claude Opus 5.5,

Introducing MCP tools for @dair_ai. Connect the MCP to your Codex/Claude/Grok Bot to discover and explore the top AI papers. We carefully curate the papers, so you can expect mostly bangers. The index

Grok 4.7 ranks first in legal matters Today we are announcing Harvey LAB-AA v1.1 in collaboration with Harvey. This updates our scoring methodology for the Legal Agent Benchmark (LAB) to add a halluci

🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷 We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level software engineering. Mistral Large 4 scores 13.3%, ahead of Muse Spar

Banger paper from Stanford and Together AI. They show why it might be a good idea to let your agent team learn its own way of working together. (bookmark it) Three models (o3-mini, Claude Sonnet 4 and

Together is #1 on @OpenRouter token share across top open coding models, today: 🥇 @Zai_org GLM 5.3 Flash: 29.2% 🥇 @deepseek_ai V4.1 Flash: 25.6% 🥇 @Kimi_Moonshot Kimi K3: 18.9% Running coding agents o

Learn to optimize your own harness, folks. You can squeeze much more performance and value from a harness. This work claims that a self-modified harness matched Codex for 4.03. Specifically, a coding

We show how you can use the River API to fine-tune open weight models to beat GPT-6 Astra and Claude Opus 5.5 on tasks you care about, with less than 1% of the cost! A few years from now most of the L

Introducing MCP tools for @dair_ai. Connect the MCP to your Codex/Claude/Grok Bot to discover and explore the top AI papers. We carefully curate the papers, so you can expect mostly bangers. The index

We were able to fine tune an open weight model (Qwen3.5-9B) on proprietary internal data at @coinbase, and it out performed frontier models, at much lower cost and speed, for a specialized task (fraud

Learn to build custom harnesses, folks! Here is an impressive paper showing the impact of a good custom harness on long-context work. GLM-5.3-Flash goes from 4.4% to 61.1% exact match on OfficeQA Pro

New open source app built with Together AI! Gloss turns your saved articles, PDFs, and X bookmarks into a beautiful personal library, with lightning-fast AI answers powered by DeepSeek V4.1 Flash. Try

Apropos NVidia’s agent trust framework, anybody remember NemoClaw? Anyone using it? Introducing the NemoClaw Deep Agents Blueprint, a reference architecture for building open agent systems developed w

This is impressive! Face-to-face conversation is where AI interfaces are heading. @tavus gave me early access to Griffin, a video-to-video model that watches and listens as you talk. It can interrupt

Remote work is so cooked If your job can be done on the other side of a screen the AI can do it better Introducing Griffin, the first model to pass the video Turing test. 48% of people who talked to i

Banger paper from Google. It's standard practice to sample several agent rollouts and trust the answers they agree on. This Google paper shows that agreement can hide shared errors, while disagreement

Thanks for sharing! We open-sourced code and models. 🤗Huggingface Paper: https://huggingface.co/papers/2609.38597 💻 Code: https://github.com/nv-tlabs/PixelUMM 🤗 Model: https://huggingface.co/nvidia/Pi

Learn to optimize your own harness, folks. You can squeeze much more performance and value from a harness. This work claims that a self-modified harness matched Codex for 4.03. Specifically, a coding

pplx-decider-v1.1-27b, our updated open weights multimodal decision model, is now available. It scores the highest on the new @huggingface Decision Index 0.3 benchmark. pplx-decider-v1.1-27b costs hal

Banger paper from NVIDIA. One exciting trend I am seeing is building verifiable environments for your agents and training the harness alongside the model. Co-evolving the harness and the model is a bi