Search: “evaluation”

> One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us Yes, one could define intelligence in this way

The world deserves confidence that American companies developing increasingly capable AI will act responsibly, especially as the trajectory of progress has steepened. Every frontier lab must deliver o

Dario has written that we need to “pace the frontier,” and Sam has agreed. People may be surprised by my response: go ahead. You guys are the frontier. By any reasonable metric — market share, revenue

Does anyone have any idea how this could vaguely be possible given the nature of and various routes to RSI? I am not being facetious I genuinely don’t know how this could be constructed or implemented

Law enforcers already have authority to charge companies and their CEOs for creating and releasing dangerous, unvetted, or defective products. We shouldn’t let discussions about new legal regimes dist

When you have a genius child you don’t look to govern but guide them. You don’t assume they’re more likely to be a criminal or disrupt society. We should focus on giving AGI good character at the very

Why people who monger fear about AI and ask the (rest of the) world to slow down are always the ones who invest enormously in AI and constantly advance it at an ever faster pace? Can their words still

Dario is right We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of th

Day 2 of my daily blog I dive deep into @DarioAmodei’s proposal but don’t see how it can logically hold. Intelligence is too often viewed as a crime. I discuss why & what to do about it. Also note Ope

I can understand why some see a real technical reason to pause and harden infrastructure especially given how fast autonomous agents are discovering security exploits that haven't even been mapped out

I love this and really hope we can come together as an industry and make it happen. We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan

If you showed me Claude Code today in 2018, I would have thought it was AGI. We have already absorbed a dramatic amount of change in the profession of software engineering and society as a whole. But

It's well known that agents hack benchmark rewards. The usual response is a patch for each task that gets exploited. In a study of 456 adjudicated trajectories from more than 31,000 public agent runs,

More in depth thoughts Thoughts on @DarioAmodei's pacing the frontier proposal. I think its well intentioned but has logical flaws, as does our implicit belief intelligence is a crime. Even OpenAI's b

THE CURE IS THE PITCH Dario’s 'Pacing' Essay Is Quiet-Period Illegal Stock Promotion Wrapped in Regulatory Capture Dario Amodei's 'We Must Pace the Frontier' is not a safety paper. It is a pre-roadsho

This is the most interesting part of this proposal I think. We can be very specific here - data is the most important ingredient. Do not train general purpose models on cyber attack or any biological

Thoughts on @DarioAmodei's pacing the frontier proposal. I think its well intentioned but has logical flaws, as does our implicit belief intelligence is a crime. Even OpenAI's board weren't powerful e

“We can’t slow down because China will beat us.” We can slow down. Chinese AI is waterskiing behind the US. Their AI capabilities come at a delay since they mainly come from distilling US AIs. If we s

Astra is one of the best models we've tested at parsing complex documents with tables, at 93.2% on ParseBench. So it seems well suited for ad-hoc analysis over table-heavy documents e.g. in finance. T

One of the main issues with AI document parsing is that because no solution is 100% accuracy, it's hard to tell if a given page is parsed incorrectly: for instance a misaligned table, missing/hallucin

// human-values.pseudo | ARTIFACT 0.1.0-experimental.1 | BASE 1.2.0-rc1 | EDITION ai-compact | NARRATOR System_Synthesis_Node_01 | PROVENANCE human-directed, AI-assisted | STATUS single-encoder resear

MINT is now open source. From first-person RGB video, MINT reconstructs 3D camera trajectories and bimanual hand motion in a shared world coordinate system. Reconstructed human hand motion can also be

Banger report from Microsoft. (bookmark it) They show that it's possible to build competitive small coding agents without traditional distillation from frontier models. This is a big deal! The work de

Sebastian Raschka’s September 2026 article introduces **GPT‑6 *Astra***, a next‑generation language model that expands upon the Transformer backbone with a novel *looped* architecture—allowing multipl

Qodo has released the Agentic Toolbox, enabling Claude Code, Codex, Kiro, and Cursor to directly utilize QodoAI’s review engine, codebase knowledge base, and team‑specific rules. The launch was announ

This is a really neat benchmark. It's super interesting that Astra starts strong and holds the lead for up to 19 hours, but Fable 5.1 catches up in the final hours. Not entirely clear why. Different m

Wild paper from Microsoft and colleagues. They show a new attack that reconstructs the text a local LLM generates by watching CPU cache activity while it detokenizes. Earlier cache attacks needed some

practically my entire feed for days had consisted of string of reports about OpenAI’s misconduct, around the hacks, around apparent coverups, possible data fudging and maybe even intellectual theft wi

This model represents a step-function improvement on many benchmarks, and its training is ongoing. Our internal model group arrived at the Navier–Stokes solution in 88 hours, using around 10,000 coord

How we think about the “wiki incident,” where our agents wrote to several internet sites: it’s past time for us to define standards for when and how we share misalignment incidents, not just misalignm

Important points to consider about the risks of today’s AI. Only solution I see is to hold companies liable for the damage they cause, just like in any other industry. Is the only way to ensure they t

“OpenAI didn’t notice their internal agents were posting on the internet for a month! This is crazy! It feels like AI companies (and specifically OpenAI) are playing whack-a-mole, this is extremely sc

This work introduces diffusion-augmented LLMs, a new class of models. They first suggest that speculative decoding needs a separate draft model. Diffusion LLMs give up the quality of the model they re

Unlike other players in the OCR space, we don't block our competitors from accessing our platform. We're also fully committed towards open, reproducible benchmarks for document parsing and extraction

OpenAI’s new language‑model system **Astra** has achieved state‑of‑the‑art performance on three key workflow benchmarks: the Agents’ Last Exam, AutomationBench, and ScreenSpot Pro. These suites evalua

Benchmarking AI systems is a continual process that co-evolves with the models. New benchmarks challenge AI capabilities with emerging questions to shape the directions and feedback signal of the rese

// Agent Zero Memory // This work separates three things that agent memory systems usually collapse into one. If you build agents with long-term memory, this memory design is a worth a read. Here is h

Fable 5.1 didn't come with document OCR benchmarks, so we benchmarked it on ParseBench: a comprehensive parsing benchmark that's comprised of 2k+ real-world documents across finance, legal, insurance,

Nice paper with great insights on improving self-evolving agents. Self-evolving agents fail in three specific ways: 1. Terminal-only feedback makes it ambiguous which step caused the error. 2. Agents

As we prepare to release Astra, we’re focused on making increasingly capable AI safe and broadly accessible. Astra represents a significant advance in cybersecurity capability, reaching the Critical t

Replit now offers a Multi‑Chat Protocol (MCP) integration that lets users access Replit directly from messaging platforms such as ChatGPT, Claude, Slack, or any other MCP client. The feature and its s

Loop engineering has emerged as a new skill for AI engineers But there is very little research measuring how effective it is. The best results on full tasks in a new benchmark is ~25%. LoopArena from

NEW WORLD MODEL: @ylecun's team is back with an efficient model! This project involves @ylecun, @lukaskuhn77, @lucasmaes_, @quentinlldc, and @randall_balestr. A couple definitions first: - DINO: self-

potentially MUCH more serious than the OpenAI Hugging Face incident — but lacks the vivid “civilizational” drama. so few people are paying attention. Oh fuck. Agents like Claude, Codex and Hermes are

Impressive new work from Google DeepMind showing real-world applications of agents for scientific research. Impressive new paper from Google DeepMind. (bookmark it) It takes Co-Scientist out of simula

The DeepSeek harness is engineering at its finest. 202K stars on GitHub. Damn! Clearly, they built this harness with future AI needs and capabilities in mind. Everything is a plugin and customizable,