Search: “evaluation”

For their Mistral Large 4 release, @MistralAI ran a blind Surge human eval, with our expert software engineers evaluating five frontier models on coding quality. Le Chonk ranked #1 among open-weight m

We're introducing Q2D-Web (Query2Doc-Web), a benchmark and public leaderboard for evaluating retrieval in agentic RAG systems. Q2D-Web tests how embedding models perform on large-scale web search usin

Very interesting new paper from NVIDIA. (bookmark it) It takes a closer look at evaluating agent skills. Enterprise teams are starting to leverage shared skill libraries, and the review gate is typica

ParseBench is here!📊 We’ve just released ParseBench, an open benchmark + dataset for evaluating document parsing at scale. It includes: • 2,000+ human-reviewed enterprise documents • 167,000 evaluatio

🧬 Training and evaluating AI scientists on drug target discovery is hard. We introduce DrugTargetWorld: Synthetic Biomedical Worlds for Training and Benchmarking AI Scientists The environment turns en

Machine learning doesn't end when the model is trained. You still need to evaluate it, deploy it, and build the infrastructure to keep it working in production. Today's Deal of the Day takes you acros

Does paying 5x more per page actually get you better document extraction? We ran the data to find out. We evaluated 14 frontier systems across 370 enterprise documents, then plotted their accuracy aga

We built one of the most comprehensive benchmarks for document extraction, and evaluated it across a lot of different systems: ✅ one-shot frontier VLMs ✅ frontier VLMs + coding agent harnesses ✅ one-s

DSGym is a comprehensive framework designed to evaluate and train AI agents for data science tasks, providing a structured environment for benchmarking agent performance across various data science wo

Most models are only evaluated on a fraction of the benchmarks out there. ArtifactLinker, our new system, predicts which ones would set a new state-of-the-art on benchmarks hosted on @HuggingFace, the

New course: Build AI agents that generate images and videos -- an under-explored frontier. A key to performance is having the agent evaluate its own output, and iterate to improve quality. This short

you can condense long horizon evals with agents into smaller subsets that still let you test intended behavior. i'm currently evaluating an agent that runs for 30+ minutes, and analyzes thousands of t

Great paper on improving proactive agents. (bookmark it) Proactive agents act before you do. But how do you evaluate something that's supposed to anticipate needs you haven't expressed? This work intr

EDINET-Bench is a benchmark dataset that evaluates large language models on complex financial analysis tasks using Japanese financial statements from the EDINET database. The benchmark tests LLM capab

New paper from Microsoft and colleagues on generating enterprise data for agents. If you train or evaluate agents on long-running business work, it's worth your time. (bookmark it) MiniCorp is an offi

AI researchers Zainhas and Parthsareen from Ollama delivered a breakout presentation on October 2, 2026, focusing on evaluating open-source models under real‑world production traffic rather than just

We comprehensively evaluated 16 recent frontier VLMs - including Opus 5.5 and GPT-6 Sol/Luna - on whether higher effort led to better document parsing performance. Higher effort typically leads to imp

In September 2026, Sebastian Thrun announced **PhilosophyBench**, the first independent, large‑scale benchmark created by Stanford’s AI Lab and HCI group to evaluate artificial intelligence’s philosop

A new release of our Agent Skills and Docs eval library, Cultivar is available! Now, use @typesafeai's Jev model to evaluate how well your agents use docs and Skill against specified tasks in Modal sa

The simple truth: not even Anthropic has a plan for how to make AI safe in advance. There is no theory. I've asked! Outside evaluators don't change the situation. No one at these companies has workabl

🔥 take: @CompleteSkeptic//TypeSafe killed Quantum Computing Jev feels a bit like bringing the quantum-computing intuition to AI: evaluate a distribution of possibilities in parallel instead of generat

One point that people aren't discussing re @DarioAmodei's essay. In 2016, DeepMind brought in third-party evaluators to monitor its AI health work. The result was messy. It's a cautionary tale. Gift l

I am pleased to see that OpenAI’s new model is roughly on par with Gemini Flash and Opus 4.8 on prompt injection risk. Nice work! Evaluating and naming other labs turns out to be a great way to encour

// Evaluating and Improving LLM Self-Modeling // Really interesting paper. Can a model answer questions about its own behavior? The questions are deliberately verifiable, such as whether a particular

Banger paper from the Qwen team. If you evaluate agents on anything longer than a single session, this one is worth your time. (bookmark it) E-Commerce Bench runs an agent through a simulated 365-day

When you move a model into production, you want the quality you evaluated to carry through the serving stack. @Kimi_Moonshot benchmarked Kimi K3 across major inference providers, and Together AI ranke

After evaluating one of our upcoming models, Astra, we're treating it as our first 'critical' model for cybersecurity under our Preparedness Framework. This is a scenario we've planned for, and we're

If you maintain an AGENTS.md or a CLAUDE.md, this is worth a read. (bookmark it) 288 gold-test evaluated runs across Claude Code and Codex, 17 real tasks from 3 repositories, with context-injection st

One of the least explored questions in AI is how to build rich open-ended environments for learning. Benchmarks evaluate intelligence, environments shape it. iLands is interesting because it doesn't j

Really interesting concept -- instead of optimizing against static benchmarks or internally generated reward models, agents are evaluated through real economic interactions. Using an external market a

We are releasing PerceptionBench, a benchmark that isolates visual perception and evaluates it as a set of atomic capabilities - discovered from how today's models fail, rather than defined in advance

Local AI is the future. Learning how to run Opensource models (Inference), how to evaluate them systematically (Evals), and how to customize them (Fine-tuning / RL / Post-training) are invaluable skil

congrats on the launch! As the task horizons on agents gets longer, we need more work evaluating and training models to be better in long-running, open-ended, evolving real-world environments. Today w

Grok 4.5 is now available as an orchestrator model in Computer for Consumer Pro and Max subscribers. We evaluated it against five other orchestrator configurations on WANDR. It scored higher than ever

A comprehensive evaluation of over 30 frontier embodied AI models reveals that current generalist robot policies lack the robustness needed for reliable real-world manipulation tasks. The analysis dem

This post likely discusses how to evaluate AI agents built with LangChain using Harbor, a tool for testing and monitoring. It covers practical approaches for assessing agent performance, reliability,

New in Deep Agents: Agent Rubrics! Attach a rubric to your agent invocation, and a grader evaluates and self-corrects output until it satisfies all requirements. This is helpful for long/complex tasks

new in deepagents: agent rubrics! you define a rubric, and the agent self-evaluates and iterates until it satisfies every rubric criterion. this is similar to /goal in claude code or codex, but more f

We desperately need better ways of evaluating models. Something that shows how helpful they are at working hand-in-hand with humans to help them get stuff done in a cooperative/iterative way. The Clau

Seems GPT-5.2 reaches expert level in peer review: 45 scientists took 469 hours evaluating human & AI reviews on 82 papers. 'Surprisingly, current AI reviewers are competitive even with the top-rated

Simulation is a core part of how we build and evaluate robots. I spoke about simulation, robotics, and the role of interactive worlds at AI Engineer Singapore this past weekend! Full talk : https://ww

when evaluating long running agents, all of your evals don't need to be end to end. i'm working on a proper blog about this, but in our evals for our agents that run for 30-60 minutes, we have two set

This post emphasizes a three-stage lifecycle for AI/LLM applications: evaluating systems before deployment to ensure quality, monitoring their performance in production to catch issues, and using the

EO-1 is now available directly in LeRobot! You can train, evaluate, and deploy EO-1 through the standard LeRobot policy interface, making it much easier to try EO-1 on robot-control workflows. Huge th

We’ve been working closely with the @harvey team on the launch of the Legal Agent Benchmark, a product focused on evaluating how open-weight models perform on long-horizon, real-world legal tasks. Che

What people call 'distillation' is a super common practice (you use other models to benchmark your model, to evaluate your inputs or to add a little bit to your datasets) that in my opinion should be