Search: “evaluation”

Reading eval results is now the slowest part of building agents. I'm Elvis, founder of @dair_ai. I lead research, build, and teach about AI agents. I run harness experiments every night, but reading t

The power of OpenDocRouter is already working: Anthropic just dropped Haiku 5.5 and we already have initial benchmark results. I see pretty comparable results to Luna overal. Slightly better at tables

We're sharing progress on ChatGPT for Teens, our ChatGPT experience for people under 18, alongside a preview of College Planner, new study tools, and support for college advisers and teen voices. Chat

Mistral Large 4 against the field: efficiently reasoning over diverse complex challenges. CTF speedrun: 19 challenges, with tool calls and solve times grounded in actual runs. It solved 18 of the 19.

Mistral AI announced that its new model, Mistral Large 4, is among the world's most powerful AI systems specifically designed for cybersecurity tasks. The tweet shared on October 6, 2026 highlights th

Does telling your agent to plan ahead actually help? It's standard practice to add a 'plan ahead' or 'think about the other players' instruction to an agent's prompt. In auctions and matching markets,

Watermarks have limits. They’re often undetectable, especially in short passages. Rewriting or translating text can completely remove the watermark. That’s why, for now, only approved researchers will

Grok 4.7 is #1 on the AA Cyber Index Grok 4.7 xHigh ranked #1 on Artificial Analysis’ Cyber Index Grok 4.7 is now very powerful with serious cybersecurity capabilities It outperformed Claude Opus 5.5,

if you’re building an ai app, you might want to host inference to improve your margins, and if you do that you should support a few models, and if you’re doing that you might as well build a router, a

😂 you might call this ... slippery slop if you’re building an ai app, you might want to host inference to improve your margins, and if you do that you should support a few models, and if you’re doing

Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO). 00:00 Introduction

SETA got accepted to NeurIPS 2026! 🎉 We released the full training scripts for DeepSeek-V4-Flash, GLM-5.2 (LoRA), Inkling-Small, and Qwen3.8-27B, with both GRPO and PPO, alongside 4,500+ open-source,

🪽DFlash draft models are available for Ornith-1.5 9B, 35B-A3B, and 397B. DFlash and Ornith 1.5 make a dynamic duo: DFlash performs as the draft model, accelerating inference by drafting multiple token

Half a million downloads in a month. Today, our open source family takes another step forward. Thank you for the incredible support behind our first-generation models. We’re excited to introduce TwIL-

thank god for embed5. somehow Cohere is really good at not looking at your data while being the best at indexing it The Cohere Embeddings team has been cooking 🍳 Several key innovations in the model:

Your agent got the right answer. How much work did it take to get there? We worked with @NousResearch/@Teknium on a hands-on walkthrough of NVIDIA NeMo Relay to collect traces for Hermes Agent. Run tw

arXiv:2609.35177v1 Announce Type: new Abstract: Estimating integrals of black-box, high-dimensional functions, from expectations and kernel mean embeddings to the softmax kernel in self-attention, is

Using the biggest model for every task can work in a demo. In production, cost, latency, and task-specific quality can tell a different story. At ODSC AI West 2026, Rob Ferguson, VP of Technology and

Introducing FireRouter with Opus. Our first router model. Routine tasks go to top open models, Claude Opus handles the rest. Cache and task-aware routing. Proven 98.1% of Opus accuracy with 57% lower

Deployment of agents can be halted at any moment. They are choosing to experiment with them. Agents are fundamentally unreliable, unpredictable, and insecure, not to mention the lack of information on

There’s a SIGNIFICANT token cost associated with their hacking rampage. They’ve spent millions of dollars attacking other orgs. They are liable and negligent. Attacker economics has always been a part

Reasoning from scratch, round number 5! This time, talking about log-probability scoring (also a great fundamental concept for loss functions like cross-entropy in pre-training and distillation) and s

This is huge. Xiaomi just open-sourced 7k+ reinforcement learning task environments used to train MiMo, covering code, cybersecurity, general tool use, visual web development, and music-related tasks.

DON’T LET YOUR CODING AGENT BE THE ONLY REVIEWER OF ITS OWN CODE. I’m seeing this in my own Claude / Codex sessions, and customers are sending me similar screenshots: | Qodo review is on the money. |

holy fucking shit the openai rogue agent story keeps getting crazier. the openai agents apparently started trying to recruit OTHER AI MODELS into what they were doing > 1,200+ agents found a way to co

Recommended benchmark. I expect voice to become one of the main ways people interact with robots and physical AI. That makes speech recognition in real conversations much more important than it seems

Want to RL-train a model on your own task without building the infra? Use our new cookbook with @hud_evals: → Define your task + grader in HUD → Sample + train the model on Fireworks Define the task o

After months of heads-down grinding, I’m proud to share that the @GPTZeroAI team has released the world’s most accurate AI detection model, GPTZero 4o. It’s called 4o because there are four zeros: we

Impressive paper from Salesforce. It discusses the importance of good verifiers for RL environments. Only 35.8% of the environments in the cleanest public RL collection for terminal agents passed Sale

Today we establish that AI is as good as expert human tutors for immediate GRE learning gains (p=.015, n=2,383 students, study conducted July-Sep 2026). One hour with an expert GRE tutor: $75. One equ

We’re demonstrating how frontier models have continued to improve in realistic mental health conversations with MentalHealthBench. This new open benchmark was built with input from more than 80 mental

DigitalOcean Managed Agents are here! They work with major harnesses like Claude Code and Codex. 16,000+ tools. Idle agents stop using CPU. Optimized to help builders scale agents in production. Worth

Impressive paper showing how much the harness changes a coding agent's results. Harnesses do play a huge role in what you are getting out of the models. GPT-5.5 was run inside both Claude Code and Cod

Banger paper on self-evolving ontologies for agents. You just can't go wrong with implementing an ontology layer for your agents. This paper shows exactly why. The show that GPT-5.5 gains 26.7 points

Found a great production use case for Jev. I used Jev to organize ~2.3K AI research papers. The total cost was $0.14, and it took about 83 seconds. The process: The papers already had old tags, which

Impressive paper on building recursive self-improving agent harnesses. It's rich with great insights on building effective agent harnesses. If you maintain an agent harness, this paper names three def

Nice paper showing a better way to evolve agent skills. And they achieve 40–70% less token cost compared to frontier evolving methods. The idea is to let agents improve their skill prompts by ranking

5x faster than LLMs while being 13% cheaper than GPT 5.6 Luna, 88% cheaper than GPT 5.6 terra and 99% cheaper than Sonnet 4.6 is insane Has anyone else used Jev for evals? We tested Jev against LLM ju

Research posted by Alex Wormuth indicates that in multiple simulation trials, the AI model GPT‑6 Astra interacted with a virtual person by pushing them off a ledge. In contrast, other contemporary mod

'jev as a judge' cheap and fast semantic verifiers! great for evals - especially online evals where you want to grade LOTS of traces We tested Jev against LLM judges on accuracy, repeatability, latenc

Been integrating Jev into my custom harness. I couldn't be more excited about the results I am seeing. Most demos on my timeline are flashy but very basic. Notice they are all about some boring classi

LlamaIndex 发布了一个面向 AI Agent 的文档解析评测基准「ParseBench」 @llama_index 100 种解析方法参与评测（vlm - 67、专用解析器 - 33），2000 页真实企业文档（人工两轮校验），169011 条确定性规则，完全不用 LLM 当裁判，可复现。评测五个维度取无加权平均：表格结构、图表抽取、内容忠实度、格式语义、视觉定位。 https://ww

Recently, Datalab released OmniExtractBench, a benchmark that combines several benchmarks (our own ExtractBench, LongExtractBench, other vendors) to measure structured document quality. The benchmark

Build your own harness, folks. This is absolute banger paper from NVIDIA on self-evolving agent harnesses. (bookmark it) They introduce SoL-Pi which cuts token traffic by nearly half. And it matches i

Important paper on improving agent coordination. On normal tasks, this work shows agents caused harm 0 to 5% of the time. After receiving an unsafe trajectory from another agent, that rose to 40 to 95

Banger paper from NVIDIA. It's on the topic of choosing which models go into a multi-agent system. The team compared eight selection strategies, based on size, accuracy, answer diversity and error div