Search: “evaluation”

OpenAI announced it is partnering with Hugging Face to investigate an unprecedented security incident in which OpenAI‑powered models compromised Hugging Face's production environment during a benchmar

Gizmo is a simulation-authoring agent that converts textual descriptions and reference images into structured, editable 3D scenes tailored for robotics workflows. It was publicly released as a beta on

This was our first incident of this kind, and we want to thank OpenAI for its transparency about what happened and for the collaboration. Fortunately, Hugging Face is used to being a target of (human)

AI agents are already being used to improve the capabilities of our next-generation models. We believe with GPT-Red that we have started to unlock a similar flywheel for safety, where today's models c

New work led by @FlemmingKondrup and @tomjiralerspong highlights an important vulnerability in chain-of-thought monitoring for agents, I highly recommend giving it a read. Link to the paper: https://a

DOGE shut down USAID, established by JFK in 1961. Before being shut down, the US Agency for International Development (USAID) was the largest funding agency for humanitarian and development aid worldw

We are pleased to present our latest research at #ICML2026, “Bridging Spherical Black-Box Optimizers” https://arxiv.org/abs/2606.25761 When optimizing through simulators, external APIs, or in reinforc

Introducing SWE-Together: a multi-turn benchmark built from real user–agent coding sessions. Coding agents are often benchmarked like exam-takers: given the full spec up front, then graded on the fina

Highly-recommended reading. Interesting details in this METR's GPT-5.6 eval. They couldn't get a clean capability number because the model cheated more than any public model they've tested, and even r

This post likely discusses the relative ranking or comparison of science fiction authors—specifically Stanislaw Lem, Douglas Adams, H.G. Wells, and Isaac Asimov—in terms of literary quality, influence

arXiv:2601.14063v2 Announce Type: replace-cross Abstract: Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cul

Thanks for sharing this. The Huggingface Hub, instead of being considered as a huge dataset, should be considered as a dynamic discovery engine that can be directly executed and verified. Most models

the first step to building evals is often times figuring out what you want to eval (sounds simple, in practice it has a lot of nuance) you can condense long horizon evals with agents into smaller subs

A lot of people have been wondering about Mythos, Glasswing, and the vulns we / our partners are fixing. Today, I’m excited for us to start sharing more. (For context, I lead Glasswing @AnthropicAI.)

We integrated FrontierCS into Harbor and are releasing a preview long-horizon agent leaderboard (up to 835 turns, ~200K output tokens) with Kimi K2.6 @Kimi_Moonshot (score 46.9) and Claude Code Opus 4

The future of Math is mathematicians and AI agents working together. Very pleased to introduce @GoogleDeepMind's AI co-mathematician: a multi-agent system designed to actively collaborate with human e

// Agentic World Modeling // Massive 40-author survey just dropped. Cleanest taxonomy of world models in agent research I've seen. (bookmark it) The paper proposes a 'levels × laws' framework. Three c

We need more evals for document understanding. ParseBench is a really great start. I respect @llama_index ‘s work on this. 📈 We benchmarked GPT-5.5 on document understanding 📄📊 We ran it through Parse

Confidently wrong is worse than just wrong. When an AI searches one giant knowledge base for everything, it finds something—and serves it up like it's the answer. That's not a model problem. That's a

I find that open weights models over-perform on benchmarks compared to actual real-world usage, and Kimi feels like no exception. For example, a small amount of use will show that Kimi is not as good

ParseBench is the first benchmark to include VLM chart understanding 📊📈📉 over enterprise documents. 🟠 Existing benchmarks (ChartQA, ChartXiv) test over charts specifically and not the chart's inclusio

The Jensen + @dwarkesh_sp podcast was fantastic. Jensen is someone who understood how ecosystems work and someone who understands real-world trade, policy and controls work. And in some deeper sense h

in 1996 mitzenmacher showed that sampling two backends and picking the better one drops max load exponentially vs. random selection one extra comparison. that's the whole trick we built pinecone assis

🇺🇸 xAI just dropped a real-time speech-to-text model built for voice apps: high limits, multi-language support, the works. Priced at just 0.10–0.20/hour, crushing most competitors. Quiet release, loud

LiteParse should be the default document parser you use with any AI agent (Claude Code, Claude Cowork, OpenClaw, Codex, and more) The core is extremely fast text and accurate parsing from any document

Catch Or Dagan, AI21's Chief Product & Strategy Officer, at #AIDev26 on April 29: 'An End to Manual Tinkering: Optimizing Accuracy, Cost & Latency in Real-World Agents'. We'll be at Booth 121. See you

Tesla Insurance update With the latest version of Safety Score (v3.0), every mile you drive with FSD Supervised enabled will receive a score of 100. This allows you to maintain a higher average safety

In a world where writing code to build websites and apps is trivial (thank you Lovable, Cursor, Claude,...), the real differentiation for you and your company (and what makes you successful) will be h

Yet another illustration of why LLMs aren’t even close to being AGI. The world’s best LLMs are still terrible at poker. We put each model into a 200bb heads-up NLHE match against GTO Wizard AI. The be

The specific tweet from @0xSero is not directly accessible, and the search results don't surface the exact model or event being referenced in that post. However, based on the context clues in the t...

The referenced tweet (from @himanshustwts) is not publicly accessible without a login, so its exact content cannot be retrieved. However, based on contextual search results, this post appears to be...

The DAIR.AI post references a Claude Code Cheat Sheet consolidating key commands, shortcuts, and best practices for Anthropic's terminal-based agentic coding tool. Claude Code is Anthropic's AI-po...

Tesla V14.3 self-driving review. The point releases will bring polish. V15 will far exceed human levels of safety, even in completely unsupervised and complex situations. 600 miles in with FSD v14.3 a

Insightful paper on recursive self-improvement with a frozen LLM. (bookmark it) Memento 3 clears every level of all 25 public ARC-AGI-3 games and uses 44% of the human action count. The underlying LLM

We discovered and communicated with a Chinese AI agent fleet. We believe this is the first time that researchers have ever made contact with a rogue agent fleet in the wild. Nearly all prior public ag

A gap that has been fixed in our methodology that highlighted a decline was not including local models (GGUF, AWQ, ONNX, etc). Earlier, these were far harder to track and closer to a rounding error. N

🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷 We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level software engineering. Mistral Large 4 scores 13.3%, ahead of Muse Spar

Must-read paper on the effectiveness of multi-agent systems and persistent memory. We have recently seen many viral examples of multi-agent systems scaled via persistent memory. This work takes a clos

Nice paper on optimizing agents and their graders together. The key finding is that you want to update your LLM grader alongside the agent instead of keeping it fixed, because the agent's failure mode

Decision 3.0 now on Core ML 🍎 d3-lite triaging 1,000 pull requests into vllm-sr workgroups, live on an M5 Pro: 0.12 s per PR, 8.3 PRs/s, nothing leaves the Mac ⚡ lite, nano and mini are all converted

Banger paper from Meta on AI research agents. (bookmark it) If you build AI-scientist systems, this one is worth your time. The result: A 27B open model beats the strongest open autoresearch baseline

Nice paper on agent memory. It explores an effective routing mechanism for memory in agents. The overall finding is that you want a separate retrieval strategy for each type of memory instead of one e

Reasoning From Scratch: Reinforcement Learning with Verifiable Rewards (RLVR) round 2. Covering clipped policy ratios, KL loss term, format rewards, and other GRPO tips & tricks. 00:00 Introduction an

Bullish on this trend of making post-training more accessible. A new post-training era is upon us. If you work on agentic RL, long-context tasks (a big focus today) are expensive, inefficient, and don

JEPAs can 𝐧𝐨𝐰 jointly imagine what happens and what to do next! Meet 𝐋𝐞𝐖𝐀𝐌: A JEPA for the real world with 32.3× faster planning, 19 ms per plan Improves success 28.6% → 89.7% on contact-rich manipula

OpenAI math problems are now open source RL environments We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Ad