Search: “benchmark”

If it can’t parse, it can’t perform. ❌ Reliable document understanding is fundamental to enterprise-grade #AgenticAutomation. We're excited to see the release of ParseBench, a new open-source benchmar

NEW: Added @OpenRouter's new free stealth model, Elephant-Alpha, to Hermes Agent, which you can now access if you run `hermes update`! I also had Hermes come up with an agentic benchmark on the fly to

EinsteinArena is an open-source benchmarking platform developed by Together AI designed to evaluate and rank AI models, with a live public leaderboard tracking model performance. The project welcomes

Grok Grok-4.20 just took the #1 spot in the world for Medicine & Healthcare on Text Arena Grok is already saving lives by identifying critical conditions that human doctors miss. There have been many

Ethan Mollick, a Wharton professor and prominent AI researcher, posted commentary on what appears to be a notable AI capability advancement, framing it as a predictable progression along established p

After seeing that Claude Mythos marketing turned out to be, as expected, a scam, I wanted to make a master list of tricks being used to market LLMs. The master list includes statements directly from l

Nous Research shared a post celebrating a model or benchmark achievement reaching the top of a performance leaderboard, likely referencing one of their Hermes or other open-source model releases. The

Yet another illustration of why LLMs aren’t even close to being AGI. The world’s best LLMs are still terrible at poker. We put each model into a 200bb heads-up NLHE match against GTO Wizard AI. The be

A study commissioned by *The New York Times* and conducted by AI startup Oumi tested 4,326 Google searches using the SimpleQA benchmark, finding that Google's AI Overviews were accurate 85% of the...

Amazon released the Nova 2 model family on December 2, 2025 at AWS re:Invent, comprising Nova 2 Lite, Nova 2 Pro (Preview), Nova 2 Omni, and Nova 2 Sonic, all featuring a 1-million-token context wi...

NEW paper from Google on multi-agent research agents. It's one of the first systems that handles end-to-end LaTeX generation, targeted literature reviews, and conceptual diagrams as a decoupled, stand

Strong release! GLM-5.1 is a DeepSeek-V3.2-like architecture (including MLA and DeepSeek Sparse Attention) but with more layers. And the benchmarks look better throughout! Looks like THE flagship open

Z.ai's GLM-5.1 is a 754-billion parameter open-weight Mixture-of-Experts model released on April 7, 2026 under an MIT license, designed as a post-training upgrade to GLM-5 with a focus on long-hori...

Simon Willison tested Z.ai's GLM-5.1 using his standard 'pelican on a bicycle' SVG benchmark, and the model stood out by spontaneously generating a full HTML page with both the SVG and a separate ...

INCREDIBLE GLM-5.1 weights are now opensource > i’ve had early access to the weights for the past few days > and yeah… this one matters a lot benchmarks? > SWE-Bench Pro: 58.4 > beats Opus 4.6 (57.3)

we’re getting used to it by now but DeepSeek calling this a “minor update” while moving Terminal-Bench 4.0 from 31.2 to 42.3 is very funny. also reports 77.0 on DeepSWE (vs 74.0 for Opus 5 and 73.0 fo

A lot of real-world documents embed extremely dense charts that require humans to not only 'perceive' the chart values, but also reason over them and reconcile with other complex elements (e.g. tables

Mistral AI introduced a tool aimed at high‑energy physics that can reconstruct particle collisions, identify particles, flag unexpected signals, and evaluate future detector designs. The announcement

GPT-Rosalind is built to support that work: helping researchers reason across scientific literature and biological data, with scientists bringing the expertise and judgment to evaluate the evidence. E

The Agentic Scientist: Orchestrating Coding Agents for Automated Scientific (Co-)Discovery Blog: https://pub.sakana.ai/agentic-scientist/ GitHub: https://github.com/SakanaAI/AI-Scientist-v3 arXiv: htt

On October 13 2026, researcher Ethan Mollick tweeted that observing Gemini 4’s behavior reveals how many other LLMs are likely distilled from Claude or GPT—and thus use a similar training pipeline—mak

Nice paper on optimizing agents and their graders together. The key finding is that you want to update your LLM grader alongside the agent instead of keeping it fixed, because the agent's failure mode

We were able to fine tune an open weight model (Qwen3.5-9B) on proprietary internal data at @coinbase, and it out performed frontier models, at much lower cost and speed, for a specialized task (fraud

Another great paper from Salesforce AI Research. The finding is that a 31B open Gemma-4 web agent scores 74.6% on the 9-app WebArena Infinity set, above Gemini 3 Flash with browser use at 70.1%. They

Today I’m excited to introduce OpenDocRouter - a unified API for document parsing with the latest frontier and open-weight models. There are a lot of VLMs and OCR models that can be used for document

Introducing Agr, an open decision model by Command Code. · Agr (31B) and Agr-flash (360M) · 58.15 on Decision Index 0.2.1 · Focus on: tool calls & routing · TypeSafe SDK support Agr is best-in-class o

Thanks for sharing! We open-sourced code and models. 🤗Huggingface Paper: https://huggingface.co/papers/2609.38597 💻 Code: https://github.com/nv-tlabs/PixelUMM 🤗 Model: https://huggingface.co/nvidia/Pi

This is impressive! Face-to-face conversation is where AI interfaces are heading. @tavus gave me early access to Griffin, a video-to-video model that watches and listens as you talk. It can interrupt

BREAKING: Grok is helping power America .gov, the U.S. government’s new AI website. 🇺🇸 U.S. Chief Design Officer Joe Gebbia says Grok & Gemini power the site. Americans can ask questions and get answe

Decision models should do more than pick a label. To truly act, they need to extract evidence, understand relationships, produce structured records, and keep related decisions consistent. And they sho

Impressive paper from Salesforce. It discusses the importance of good verifiers for RL environments. Only 35.8% of the environments in the cleanest public RL collection for terminal agents passed Sale

Banger paper from Stanford and Together AI. They show why it might be a good idea to let your agent team learn its own way of working together. (bookmark it) Three models (o3-mini, Claude Sonnet 4 and

Exciting work from NVIDIA. (bookmark it) Interesting to see this approach to turn public Agent Skills into RL environments. Lots of excitement around RL environments so this is a great read. Skill2Env

glance-vlm speedlab is now open source! read: https://glance.yohei.me/speed/ try: https://github.com/yoheinakajima/glance-speedlab turn any webcam into multiple live AI detectors (emotion, count, obje

Introducing Qwen Intelligence, bringing personal intelligence within everyone's reach. 📱✨ It launches with three SOTA agents: 🥳 - Mobile Planner Agent: plans, decomposes & orchestrates complex tasks.

Impressive paper showing how much the harness changes a coding agent's results. Harnesses do play a huge role in what you are getting out of the models. GPT-5.5 was run inside both Claude Code and Cod

Opus 5.5 is the best frontier model for parsing tables in PDFs. We ran it through ParseBench, and it scored 93.9%, a 7%+ increase over Opus 5, and beating Fable/Gemini/Astra. It's comparable to LlamaP

the design of http://evaluatorbench.com grew out of a summer project, http://epistemedia.org (which i still need to write about), and runs on @activegraphai the pattern across all three projects is th

The essay below from Aaronson is just unmoored from reality, sorry. It is just not true, if you try to approach the question in a data-oriented fashion, rather than via vibes and speculative fiction a

DeepSeek V4.1 Flash is now available on Together AI’s model hub (https://www.together.ai/models/deepseek-v4-1-flash). The release includes the DeepSeek V4.1 Flash API with updated pricing, performance

agreed, one of the biggest bottlenecks right now is my own ability to internalize the information that these models convey the latest models (astra, fable 5.1) are either way too verbose, or try too h

Experiences people can interact with in real time. A new kind of foundational moment. A sneak peek into what's possible and what's coming for real-time video generation. We believe instant generation

The new DeepSeek V4.1 Flash model is mindblowing - back on top of the open-source model leaderboard and extremely cheap. It has a lot of very smart ways to be efficient and highly capable so I made a

Astra is awesome, but if you're using it specifically to parse and extract a massive volume of documents, you will break the bank (10c+ per page on parsing and extraction!) We've added a native LlamaP

Machine learning doesn't end when the model is trained. You still need to evaluate it, deploy it, and build the infrastructure to keep it working in production. Today's Deal of the Day takes you acros

Is the @METR_Evals long horizons measure effectively saturated? No updates to the most famous graph of AI progress since May, but other published work suggests you can effectively get 18+ weeks of wor