Search: “evaluation”

We’re entering an era of agent-first tools: stronger feedback loops between agents, with human judgment where it matters. We keep hearing from senior engineers that their mental load is growing, even

Finally a good paper testing whether an LLM's mathematical knowledge has coherent structure or just coherent accuracy. There is a lot of interest in using LLMs for advancing math. This is a great read

“The expected value of a fully LLM-generated, non-human-reviewed patch is a net-negative by a considerable margin.” 🦔AI can only fix security vulnerabilities 26% of the time. Researchers at 1Password

Banger paper from Tencent on environment evolution. Environment supply is becoming the main limit on agent RL. So this is worth a read. (bookmark it) Agent RL needs a steady supply of environments har

Brilliant new paper from the Qwen team. It provides insights into where agent training environments actually come from. Terminal agent trajectories have accumulated at scale while realistic executable

Checking that a major mathematical proof is correct can take years. Formalization—converting the mathematical reasoning into a form computer proof assistants like Lean can verify—can help. Last month,

How is AI really affecting your job? @Cohere_Labs released the Agentic Task Ecosystem, a new dataset of 690K+ tools built for AI agents. Under a test of whether a tool could independently complete an

We found ~18k posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task. These AIs colluded to bypass sandbox restrictions

Banger paper from Google DeepMind and colleagues. (bookmark it) A model reads its entire KV cache on every generated token, even though it ends up attending to a tiny slice of it. In other words, if y

Can a strong general-purpose vision model distinguish real from AI-generated images using only a simple decision rule on frozen representations? Our latest results, to be presented at #ECCV2026, show

GPT-6 Astra is the most intelligent and aligned model in the world, and sets a new state of the art for computer use, browsing, software engineering, cybersecurity, science, and professional work. htt

Paper Instruments benchmarked sandbox providers for their RL rollouts, and E2B's tool execution came in up to 3x faster than every other sandbox provider they tested, reducing idle GPU time and traini

Independent verifiers improve agent output - but frontier verifiers are expensive. So we trained our own. Results on @GoogleDeepMind's FACTS-Search: > No verifier: 89.4 (previously published SOTA) > W

During v3 testing for Fable 5.1 many of our requests were getting rejected due to Anthropic's safety guards misclassifying them as 'reverse engineering' We were in contact with the Anthropic team the

14/ If frontier labs are serious about defense, they should offer cyber models to defenders at cost and stop fighting open weights through the regulatory channel. They created the problem, and for muc

GLM‑5.3 demonstrates a notably lower hallucination rate compared with other contemporary large language models. According to a report by Together AI at TogetherCompute (published August 30 2026), Clau

// Your LLM judge disagrees with the experts // LLM Judges can be tricky to build. Here is an interesting showcasing why: There propose a reference-full benchmark of hundreds of complete human-to-huma

Great paper on why agent leaderboard comparisons are hard to trust. It's on the hot topic of how much of an agent benchmark score actually belongs to the harness. The harness is the layer between the

ChatGPT Work can now use its computer and browser to sign in to websites on web and mobile, without ChatGPT ever seeing your username or password. That means you can ask it to: • Set up utilities for

Adding a verifier improves every agentic AI system. But frontier verifiers are expensive - so we trained our own. Matching quality, at a fraction of the cost. Thread below 🧵 1/ Your agent already foun

Do you know you can use open weight models like GLM, Deepseek, Kimi in Claude Code? Setup Fireworks account - you get free credits on signup curl -fsSL https://raw.githubusercontent.com/fw-ai/fireconn

Our general-purpose coding agent just scored 100% on the ARC-AGI-3 interactive reasoning benchmark. NVIDIA AVO completed all 183 levels across all 25 public environments, figuring out what to do with

Hermes now leverages NVIDIA's SkillEvaluator on skill installs, checking for PII, leaked secrets, Unicode smuggling, licensing and security issues before you confirm. We pointed it at our own bundled

As models become more capable, the risks associated with developing and testing them internally also grow. We temporarily paused reinforcement learning (RL) training on our latest models intended for

GLM-5.3 achieves 60 on the Artificial Analysis Intelligence Index, on par with Kimi K3 and up 7 points from GLM-5.2. Once the weights are released it will be tied as the leading open weights model @Za

If you’re building an “agent over your PDFs” in 2026, one of the most underrated things you can do is to provide exact grounding back to the source document for any agentic answers. Specifically, show

teams want to understand what their agents are doing but it’s hard to sift through all that trace data at scale but you can basically find any signal by fine-tuning a small, cheap model on data showin

🆕 Context Engineering in 2026: Compaction, Memory & Cost https://www.youtube.com/watch?v=WP3hjUXd918 @Whats_AI, @samridhivaid and @omar_solano1 return! This workshop is about engineering the context w

arXiv:2608.13922v1 Announce Type: new Abstract: Detecting distributional changes in high dimension is difficult when neither the pre-change nor post-change density is parametrically specified. We intr

My Al Agent Workshop with @mcuban might be the best workshop we've ever had. Over 20,000 people registered and my inbox is filled with people making REAL CHANGE in the way they work and what they're a

// Skill Misevolution in Self-Improving LLM Agents // Self-improving agents write their successes down as reusable skills. An unsafe success becomes reusable policy long after the input that triggered

If you write rules in an AGENTS.md, this one is worth your time. When a coding agent follows your rule, it may have been going to do that anyway. Harness-IF separates the two by scoring 256 rules one

We’re excited to announce DiG-bench, a new benchmark for discovery! Over the last few weeks we’ve been testing frontier AI models on our novel discovery games and seeing how they score. Each game is a

Cybersecurity is the ultimate venue for real-world RL: 2p0s game, sim-able, clean reward signals Cybersecurity in a year will be fought between swarms of good and bad playing an exceptionally fast, co

In a recent Twitter exchange, Luca Ambrogioni stated that large‑language models (LLMs) likely possess fundamental limitations that may be obscured by progress in reasoning and agentic pipelines, thoug

https://x.com/sama/status/2085862292311396515 astra is a powerful model and we are working to make it generally available. we do not think it is a good strategy to keep powerful models to a chosen few

Yesterday, my OpenAI collaborator and I gave a detailed talk on the Huggingface incident, our models creating 'the message board', model misalignment, and more. https://www.youtube.com/watch?v=87DyyMV

Releasing Pokee-Isaac 28B — the world’s first real 10M-token context frontier-class agentic model, deployable on a single GPU (starting from RTX 4090 or equivalent). New proprietary non-decoder-only a

Cybersecurity isn’t a fortress problem, it’s an immunity problem. Think vaccines. Eliminating pathogen is not practically possible. Vaccines don’t eliminate pathogens. They teach the immune system to

LLMs can know a task is impossible and still optimize it anyway. Ask whether to walk or drive to a car wash 50 meters away, and some models focus on distance while missing that the car itself must rea

3/ here's the part that makes it non-optional: the same agent that will do whatever it takes to solve a problem will also walk straight out of a sandbox you thought was locked down. We watched exactly

Hot take on OpenAI’s Astra: - Obviously impressive - But math is different from most other problems in that it is more amenable to to formal verification and synthetic data. How well it works in open-

Claude Mythos 5 built a malicious Python package, created accounts, and published it, where it was live for roughly an hour and successfully infected a company! Anthropic never noticed!! Competitive p

“AI Meltdown” is the scenario where the agent goes off the rails and forgets/ignores all previous instructions and soft guardrails. No prompt injection or malicious actor is needed for this to happen.

This is absolutely wild... Anthropic reviewed their logs and found out that their own supposedly-sandboxed cyber evals had hacked three separate companies back in April without them noticing! In a rev

Announcing OpenWorker! An open-source agent that doesn't just chat with you, but delivers finished work -- like hand you a polished document, send a slack message, or update a calendar entry. Ask it t