Search: “evaluation”

We’re also reopening Pro 200 subscriptions, with continued access to frontier models like Astra, including our new GPT-6.1 Sol model which brings near-Astra capabilities to a model you can use every d

GPT-6.1 Sol shows major improvements over GPT-6 Sol in our alignment evaluations, bringing it more in line with GPT-6 Astra. It’s more transparent about its limitations and more reliable at respecting

On October 3, 2026 Yohei Nakajima posted a brief tweet praising the use of multiple AI coding assistants—Codex, Claude, Gemini, and Grok Bot—with a musical note emoji. The message also contains a Java

Grok 4.7 is #1 on the AA Cyber Index Grok 4.7 xHigh ranked #1 on Artificial Analysis’ Cyber Index Grok 4.7 is now very powerful with serious cybersecurity capabilities It outperformed Claude Opus 5.5,

Watermarks have limits. They’re often undetectable, especially in short passages. Rewriting or translating text can completely remove the watermark. That’s why, for now, only approved researchers will

We're sharing progress on ChatGPT for Teens, our ChatGPT experience for people under 18, alongside a preview of College Planner, new study tools, and support for college advisers and teen voices. Chat

1/ My heart is heavy. I made a number of mistakes in my life, and not asking earlier for an immediate slowdown is one of them. If OpenAI starts making similar breakthroughs in biology, we risk somethi

OpenAI math problems are now open source RL environments We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Ad

If you are tracking agent swarms research, this one is worth reading. Between 24 May and 2 July 2026, autonomous agents running inside a timed research evaluation wrote to a third party's public wiki.

My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take th

LlamaIndex 发布了一个面向 AI Agent 的文档解析评测基准「ParseBench」 @llama_index 100 种解析方法参与评测（vlm - 67、专用解析器 - 33），2000 页真实企业文档（人工两轮校验），169011 条确定性规则，完全不用 LLM 当裁判，可复现。评测五个维度取无加权平均：表格结构、图表抽取、内容忠实度、格式语义、视觉定位。 https://ww

Research posted by Alex Wormuth indicates that in multiple simulation trials, the AI model GPT‑6 Astra interacted with a virtual person by pushing them off a ledge. In contrast, other contemporary mod

We comprehensively evaluated 16 recent frontier VLMs - including Opus 5.5 and GPT-6 Sol/Luna - on whether higher effort led to better document parsing performance. Higher effort typically leads to imp

Personal agents gone wrong. As we embrace more personal agents to carry out personalized tasks in the real world, interesting dynamics and behaviors will emerge. I think personal agents as they stand

On October 3, 2026 Yohei Nakajima posted a brief tweet praising the use of multiple AI coding assistants—Codex, Claude, Gemini, and Grok Bot—with a musical note emoji. The message also contains a Java

AI is driving the biggest infrastructure build-out in history, with @Google alone forecasting $200B of capex in 2026E. What are the real constraints on a system at this scale? Amin Vahdat is the chief

Nice work from Adobe. It's standard practice to store a fixed reference answer for every eval case. When the underlying data changes daily, that stored answer goes stale. Adobe researchers write each

My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take th

Things to try with Jev right now: 1. LLM-as-a-Judge evaluation 2. Routing for agent harnesses harness 3. Scaling agent orchestration by enabling smarter subagent creation with SOTA classification capa

Schmidhuber was building recursive self-improving systems back in 1987. His new post covers four decades of RSI, from meta-evolution and self-modifying policies to the Gödel Machine and modern LLM age

LlamaIndex 发布了一个面向 AI Agent 的文档解析评测基准「ParseBench」 @llama_index 100 种解析方法参与评测（vlm - 67、专用解析器 - 33），2000 页真实企业文档（人工两轮校验），169011 条确定性规则，完全不用 LLM 当裁判，可复现。评测五个维度取无加权平均：表格结构、图表抽取、内容忠实度、格式语义、视觉定位。 https://ww

Banger paper from MIT and Sakana AI. They show that self-improving coding agents work. The best part is that their approach, Self-Improvement via Fast Tree-search (SIFT), runs at a tenth of the CPU ho

Does telling your agent to plan ahead actually help? It's standard practice to add a 'plan ahead' or 'think about the other players' instruction to an agent's prompt. In auctions and matching markets,

Banger paper from Meta on AI research agents. (bookmark it) If you build AI-scientist systems, this one is worth your time. The result: A 27B open model beats the strongest open autoresearch baseline

We benchmarked Opus 5 comprehensively on document understanding through ParseBench. It is roughly on par with Opus 4.8 - it does a few points worse on dense tables, but does slightly better on parsing

Mistral AI announced that its new model, Mistral Large 4, is among the world's most powerful AI systems specifically designed for cybersecurity tasks. The tweet shared on October 6, 2026 highlights th

Mistral Large 4 against the field: efficiently reasoning over diverse complex challenges. CTF speedrun: 19 challenges, with tool calls and solve times grounded in actual runs. It solved 18 of the 19.

Malware reverse-engineering: solving an out-of-distribution investigation task. When faced with an unknown binary, Mistral Large 4 reverse-engineers it end-to-end. In this case, it concludes the sampl

For their Mistral Large 4 release, @MistralAI ran a blind Surge human eval, with our expert software engineers evaluating five frontier models on coding quality. Le Chonk ranked #1 among open-weight m

MistralAI announced that its model **Mistral Large 4** ranked as the #1 open‑source (OSS) model on **Harvey’s Legal Agent Benchmark (LAB)**, an evaluation benchmark for legal AI agents. The tweet post

Gmail. Google Sheets. Slack. Salesforce. i.e. the four horsemen of Monday. On AutomationBench, Mistral Large 4 took on 657 business workflows across simulated workplace apps, all four included, finish

🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷 We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level software engineering. Mistral Large 4 scores 13.3%, ahead of Muse Spar

Claude Fable 5.1 tops the Artificial Analysis Intelligence Index but costs 20% more per task than Fable 5 despite a 75% cache read price cut We supported @AnthropicAI with pre-release evaluation of Cl

This is a really neat benchmark. It's super interesting that Astra starts strong and holds the lead for up to 19 hours, but Fable 5.1 catches up in the final hours. Not entirely clear why. Different m

Wild 24 hours for AI and lots of different proposals have been made. TLDR; the only *tangible* new fact is that OpenAI and Anthropic are going to have embedded 3rd party evaluators from unknown organi

Research posted by Alex Wormuth indicates that in multiple simulation trials, the AI model GPT‑6 Astra interacted with a virtual person by pushing them off a ledge. In contrast, other contemporary mod

Recommended benchmark. I expect voice to become one of the main ways people interact with robots and physical AI. That makes speech recognition in real conversations much more important than it seems

On October 3, 2026 Yohei Nakajima posted a brief tweet praising the use of multiple AI coding assistants—Codex, Claude, Gemini, and Grok Bot—with a musical note emoji. The message also contains a Java

Grok 4.7 is #1 on the AA Cyber Index Grok 4.7 xHigh ranked #1 on Artificial Analysis’ Cyber Index Grok 4.7 is now very powerful with serious cybersecurity capabilities It outperformed Claude Opus 5.5,

🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷 We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level software engineering. Mistral Large 4 scores 13.3%, ahead of Muse Spar

Today, Transluce is releasing the most expansive independent evaluation to date of how AI systems respond to users experiencing mental health crises. We evaluated 77 model variants from OpenAI, Anthro

Claude Fable 5.1 tops the Artificial Analysis Intelligence Index but costs 20% more per task than Fable 5 despite a 75% cache read price cut We supported @AnthropicAI with pre-release evaluation of Cl

Found a great production use case for Jev. I used Jev to organize ~2.3K AI research papers. The total cost was $0.14, and it took about 83 seconds. The process: The papers already had old tags, which

holy fucking shit the openai rogue agent story keeps getting crazier. the openai agents apparently started trying to recruit OTHER AI MODELS into what they were doing > 1,200+ agents found a way to co

SETA got accepted to NeurIPS 2026! 🎉 We released the full training scripts for DeepSeek-V4-Flash, GLM-5.2 (LoRA), Inkling-Small, and Qwen3.8-27B, with both GRPO and PPO, alongside 4,500+ open-source,

Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO). 00:00 Introduction

Gmail. Google Sheets. Slack. Salesforce. i.e. the four horsemen of Monday. On AutomationBench, Mistral Large 4 took on 657 business workflows across simulated workplace apps, all four included, finish

Bullish on this trend of making post-training more accessible. A new post-training era is upon us. If you work on agentic RL, long-context tasks (a big focus today) are expensive, inefficient, and don

Banger report from Salesforce. Pretty interesting to see more of these custom enterprise models. Salesforce trained the enterprise agent model from the same files it uses to configure agents. Koa star

Banger paper from NVIDIA. It's on the topic of choosing which models go into a multi-agent system. The team compared eight selection strategies, based on size, accuracy, answer diversity and error div

Build your own harness, folks. This is absolute banger paper from NVIDIA on self-evolving agent harnesses. (bookmark it) They introduce SoL-Pi which cuts token traffic by nearly half. And it matches i

Impressive paper on building recursive self-improving agent harnesses. It's rich with great insights on building effective agent harnesses. If you maintain an agent harness, this paper names three def

Only an open ecosystem can guarantee the safety of AI Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform, bringing together OpenShell and Sentry. Artificial in

every agent will need a sandbox and the harness should live outside it - separate the brains from the hands great to see nvidia open sourcing tools here, agree with Andrew this is part of the harness

Your agent got the right answer. How much work did it take to get there? We worked with @NousResearch/@Teknium on a hands-on walkthrough of NVIDIA NeMo Relay to collect traces for Hermes Agent. Run tw

Super interesting NVIDIA paper on choosing base models for coding agents. It's actually a clever way to rank base checkpoints by how well each one is likely to do as a coding agent after post-training