Search: “benchmark”

The AI research community released a set of notable papers between September 28 and October 4, highlighted in a weekly roundup. Key contributions include JAZ, CASD, Agensh, Jev‑Mem, AutoGym, Taste‑Ben

.@OpenRouter co-founder Alex Atallah, in his first podcast since Stripe acquired the company, joins @Replit co-founder Amjad Masad and a16z's Erik Torenberg on why the future of AI is independence and

AI researchers Zainhas and Parthsareen from Ollama delivered a breakout presentation on October 2, 2026, focusing on evaluating open-source models under real‑world production traffic rather than just

New paper from Microsoft on compressing agent context at test time. If your agent gets worse as its interaction history grows, this method is worth a look. FOCUS asks which past interactions the agent

The BEA's new Q2 GDP-by-state was released, but the key data is entirely in an infographic, without a text-layer. LlamaParse's agentic parsing tier understands documents like this by default, and fait

Remote work is so cooked If your job can be done on the other side of a screen the AI can do it better Introducing Griffin, the first model to pass the video Turing test. 48% of people who talked to i

The critical distinction between base LLMs (2024 and earlier) and modern LRMs is not symbolic tool use. It's the switch from a transductive paradigm (intuit the answer to the query) to an inductive pa

Independent numbers from @ValsAI. Fewer tokens per turn means lower cost and faster agent loops on every call. Try Ember-1 on Fireworks Serverless: https://fireworks.ai/models/fireworks/ember-1 Firewo

Lots of discussion out there about our next model(!), so I wanted to give an early look as soon as possible. Introducing Gemini 4 Argon! It shows frontier performance in complex workflows, cyber defen

PDFs aren't built for machines: text shows up as glyphs with coordinates, tables as line segments instead of table structures. @jerryjliu0, CEO of LlamaIndex, digs into what that means for AI agents i

Timelines just got shorter... AI will understand the physical 3D space better than humans by November* Is Google so back? Their new Gemini 4 Argon understands the physical 3D world better than any oth

On September 29, 2026, Twitter user Severin Field (@sevdeawesome) posted that an AI‑generated video from Claude 5.5 Opus responding to the phrase “AI is a normal technology” was the best AI‑created vi

Claude.ai achieved roughly a three‑fold speed increase in just two weeks by implementing targeted engineering optimisations. The blog post on claude.dev explains the key changes—such as streamlined co

OpenAI announced the launch of **dots**, an always‑on agent system powered by the GPT‑6 Astra model. According to OpenAI, dots are “remarkably capable” agents designed to handle a wide range of tasks

Apropos NVidia’s agent trust framework, anybody remember NemoClaw? Anyone using it? Introducing the NemoClaw Deep Agents Blueprint, a reference architecture for building open agent systems developed w

Introducing FireRouter with Opus. Our first router model. Routine tasks go to top open models, Claude Opus handles the rest. Cache and task-aware routing. Proven 98.1% of Opus accuracy with 57% lower

who's got two thumbs and just ranked #6 of 49 on image-jevbench? 👍 this guy 👍 one of the fastest in top 10, accuracy is close. calibration of confidence is where it's weakest. @airesearch12 says a sim

Elliot Arledge released a video titled “Claude Opus 5.5 just one‑shot,” which explains how to optimize CUDA kernels using a music‑video format. The clip was posted on September 27, 2026 and has accumu

🛠️ MiMo-V2.6 update: tool-call repetition, diagnosed & fixed. After the MiMo-V2.6 series models launched, we noticed them sometimes repeating identical or highly similar tool calls — burning context a

This is brilliant. Recommended reading. It's a great example of how you could potentially build more effective harnesses using System One and System Two models. It explores Jev's potential as a fuzzy

People on X: “CLAUDE IS BACK, baby!” Listen, Opus 5.5 is incredible and probably my favorite overall model right now, but it is still ridiculously verbose and over-justifies and over-qualifies everyth

Xiaomi has released an open‑source reinforcement‑learning framework called MiMo (version V2.6) that bundles a suite of simulated environments together with training scripts. The code is hosted on GitH

Banger paper from Microsoft and colleagues. We talk about human taste being important in this AI era. But for recursive self-improvement, an agent's taste also matters. The big question is: How often

Own your harness, folks. I feel like System One models take custom agent harnesses to a whole different level. It's not a surprise. This comes down to needs that haven't been met with System Two model

We announced our RSI Lab earlier this year: https://sakana.ai/rsi-lab/ Over the last two years, we have systematically shipped the foundations for autonomous R&D: ▪ LLM²: AI automating research to inv

After months of heads-down grinding, I’m proud to share that the @GPTZeroAI team has released the world’s most accurate AI detection model, GPTZero 4o. It’s called 4o because there are four zeros: we

Banger paper introducing Jev-as-a-Judge. The overall finding is that you want to use a cheap judge for most of your evals and send only the uncertain calls to a frontier model. This paper measures how

Interesting new paper from Google DeepMind. Studies agents that follow bad advice from users. Users often suggest a fix that sounds right and is wrong. XYEval adds one confident, misleading hint to ta

Just trained Tev1 0.8B, a tiny Jev-like classifier. Here it is running completely locally on my mac with @ollama & classifying some tasks. It's extremely fast: only ~50ms E2E latency. Video is not spe

Interesting paper on agent memory stored as a linked markdown wiki. Lots of great ideas and insights if you work with LLM Wikis. Wikis are useful for agents because each page holds dense text and the

Introducing Contrastive Language Model (CLM): an ultra-fast System One Model trained with a contrastive learning objective that connects states and actions. CLM-8B is pre-trained on internet-scale dat

New paper! 🫡 We introduce Matryoshka Attribution, a new attribution method which uses gradient descent to find which parts of a neural network are responsible for a behaviour. MAttr is #1 on the Mecha

Did a few quick tests and these seem very very legit. 2.6 Pro in particular is doing well in some pretty hard tasks. Introducing Xiaomi MiMo-V2.6 — Pro & Flash. Frontier intelligence, all the modaliti

Banger paper on self-evolving ontologies for agents. You just can't go wrong with implementing an ontology layer for your agents. This paper shows exactly why. The show that GPT-5.5 gains 26.7 points

Finally, a company brain that actually works! Codos starts with the people doing the work. It interviews employees to find which tasks can be automated, then deploys the automations and keeps learning

Grok 4.7 places @SpaceXAI as third, after Anthropic & OpenAI, for agentic coding. When factoring in that Grok is significantly faster & lower cost, it’s a great choice for your everyday workhorse. Gro

Jev and the System One Model: RLCD, intelligence/, reliable AI, & the end of chat-first AI https://www.latent.space/p/jev @typesafeai CEO @CompleteSkeptic explains why AI can solve extraordinarily har

MiMo-V2.6-Pro debuts as the top open weights model on the Artificial Analysis Intelligence Index (46). At 0.13 per Intelligence Index task, it lands on the Intelligence vs. Cost per Task Pareto fronti

StepFun’s new Step 5 Preview model is impressive! Had a chance to test it early. I've been testing it as a coding agent. It’s on the Pareto frontier for cost vs. capability. It is a very capable model

We're excited to announce our AI Engineer New York speaker lineup! Presented by @arizeai CEO @ExaAILabs CTO @tryramp CEO @ValsAI CEO @RogoAI CTO @modal CEO @trybasis CEO @SonarSource CEO @runwayml CTO

A new release of our Agent Skills and Docs eval library, Cultivar is available! Now, use @typesafeai's Jev model to evaluate how well your agents use docs and Skill against specified tasks in Modal sa

Banger paper from Google Cloud AI Research. If you follow autonomous research agents, this one is worth your time. ScientistTwo takes a problem from a human expert and runs the full discovery cycle wi

Banger paper from MIT and Sakana AI. They show that self-improving coding agents work. The best part is that their approach, Self-Improvement via Fast Tree-search (SIFT), runs at a tenth of the CPU ho

One of the craziest use cases I’ve found for Jev: verifiers. I am so excited about this that I at least wanted to share the high-level idea. I used Jev to build a custom verifier for the /goal feature

deepseek v4.1 flash on together ai is leading on ttft for pre-warmed queries for agent workflows, that startup time shows up again on every model turn, so shaving it down matters more as the loop gets

Important paper on improving agent coordination. On normal tasks, this work shows agents caused harm 0 to 5% of the time. After receiving an unsafe trajectory from another agent, that rose to 40 to 95