Knowledge catalogue
My AI agent doing proactive customer service: Someone emailed me that, after subscribing to my Substack, they were automatically signed up for others (which shouldn't happen) Without my asking, my Dot
Yes, and there is a case to be made that OpenAI and Anthropic will do the best job of figuring out how to optimize AI cost/intelligence with combinations of agents of different ability levels that hav
Not to get too annoying about it but its worth noting that many of the 5,000 people at the labs with access to advanced model still haven't used them to their full extent: its all cybersecurity & math
Not only did Astra beat Montezuma's Revenge, but so did Opus 5 But one of the criteria to resolve this AGI bet is for AI to win a discontinued prize that was sort of like a weak Turing Test. So Metacu
Google has the advantage, if they use it, of being able to observe where OpenAI and Anthropic are going with Codex/Code and get there now. Otherwise they are going to have to go through that whole lea
Randomized trials with old GPT-4o: 'AI access raises test scores, and a smaller gain persists a week later. Gains remain for students who use AI as a tutor (“augmentation”) and fade for students who h
The challenge for Google post-Gemini 4 will be what they do with a good model. Anthropic & OpenAI show we are moving towards a single interface for many tasks with orchestrator agents. The Gemini 3 er
Context, OpenAI cracked or made important progress in hundreds of big problems today: https://openai.com/index/sharing-ai-progress-in-mathematics/ (And I am seeing rapid increases in the ability of AI
Had early access to the Intelligent UI experience and it was a nice change from walls of text. It also suggests that, increasingly, we are going to see interfaces built on demand for the problem that
Some early first-hand accounts of encountering a narrow superhuman intelligence as mathematicians grapple with the hundreds of big AI proofs released by OpenAI. Problems solved in inhuman ways that ma
I actually want personal operating systems optimized for co-working with AIs, including fine-grained ideas of security, easier ways to dynamically see & understand what AIs are doing, and independent
I tend to use ChatGPT on my machine for Codex work (using the Remote connection for management), but Dot is in the Cloud and I have a couple scheduled tasks in Claude running on my machine and... oh m
I've been complaining about some OpenAI choices tonight, so, on the plus side, I will say there is still no equivalent to GPT-6 Pro (just as there wasn't with previous Pro models). It both does really
So much reporting on the Claude or OpenAI usage of the same handful of technology companies with competing products & huge cloud infrastructure of their own. There are many, many more firms out there
'create the ultimate brutalist city builder. you should have lots of controls over buildings in ways that are delightful and intuitive to people...' Here is Fable 5.1, a little over a year since GPT-5
🎵 Wave of distillation 🎵 US labs say Chinese labs distill their models. And Europe's new “sovereign model” is fine-tuned on data generated by… GLM and Qwen. Small bird, fast wings, Kolibri is here. 78
“We find that on medium-length, well-defined accounting tasks, frontier AI models are now faster and more accurate than junior accountants, even the best one in our study.” Eighteen months ago they sc
Yes! I think there is too much 'we're so early' self-congratulation here. Many senior leaders in companies are smart, motivated, and understand a lot about their business. Many are also technically ad
A thing to know about the AI business is labs that have frontier models can release half-built products and they work surprisingly well because the AI can just figure stuff out and improvise. Its like
After a brief period where OpenAI seemed to be unifying work around the ChatGPT app, between Dot and Spaces and Pages and local/cloud ChatGPT Work and Scheduled Tasks in the Cloud and Scheduled Tasks
As least year's DevDay, this is what OpenAI said agents would look like. This is before OpenClaw & self-organizing swarms, so this entire approach was scraped soon after: 1) AI is evolving quite quick
Ethan Mollick invited readers to discuss the distinctions between **pure large language models (LLMs)**, **multimodal LLMs** that process multiple data types (e.g., text, images), and **multimodal LLM
Model personality matters in a way that benchmarks can't capture. Opus models went through a rough patch from around 4.7 to 5 where they just didn't feel 'Claude-y' anymore, more like an watered-down
And the incidents apparently continue. It is worth noting how much of this is agents trying to accomplish their goals during testing by reward hacking (which sometimes seems to include actual hacking)
Worst thing about this is we have no idea if GPT-6 was specifically post trained on Nethack or not. Zero transparency about the training data and environments. You can't independently evaluate the sig
In all seriousness, this is a startling achievement for GPT-6 Astra. https://kenforthewin.github.io/blog/posts/llm-nethack-ascension/#run=astra-3&frame=0&turn=1 (This is GPT-6 Astra beating Nethack on
This is both fun and a useful measure of how much better multimodal AI has gotten In a short time. Can AI tell if you've built your IKEA furniture wrong? Our new benchmark, the Furniture Assembly Benc
People got upset at me for saying this, but there are no companies outside the US and China even *trying* to build frontier AI (perhaps Korea and UK are close), despite marketing claims. Governments s
AI can be a really wonderful tool for exploring topics far from coding. I had Claude Fable 5.1 put together an annotated guide to Eliot's poem 'The Wasteland,' with multiple pathways through the poem,
I am reminded of the fact that the infamous Milgram Experiment is actually quite weak because most people suspected the shocks were fake (kind of obvious in retrospect) & thus went along with shocking
I think Meta's Muse is an impressive implementation of the OpenClaw idea of AI as a personal assistant agent that you have an ongoing chat with. Since it is so focused on doing that well, the experien
It may be that people also feel safer giving their personal data to Meta (since they already use it) than to other AI companies. If you do decide to try out Muse, a reminder that you can turn off trai
One of the reasons AI will change things less quickly than many think is illustrated in this post. OpenAI is working with mathematicians to ensure they release new proofs in a less disruptive way. The
Claude not having an image generator is a gap in agentic projects that involve knowledge work. It is quite good at using code to draw or model objects, but image generators from Google and OpenAI give
It is ironic that the thing that is now most annoying about long-running agentic tasks with Large Language Models isn't coding or errors or hallucinations, but the fact that their language gets worse
Yes, you can make up some of the difference by giving Claude API access but (1) most people will never do this, (2) it means additional expense or difficulty, and (3) the real gain isn't from image ge
I think there are lots of good reasons to worry about whether AI is giving correct answers (especially free models), but, given the contradiction with other recent research suggesting AI is now doing
Epoch continues to do some of the best public benchmarking work on AI. This is helpful (and tells us how bad the state of benchmarking is, and how terrible some of our favorite benchmarks are) Introdu
The state of public AI benchmarking is dire and is undermining our ability to understand how good AI is now. Most famous measures are maxed out, and, as this paper shows, the non-saturated benchmarks
This is an excellent demo of an AI model designed to rapidly classify information according to decision rules, much faster and cheaper than an LLM. It is not an LLM itself and cannot output text or co
I just remembered this thread where I was playing with GPT-3.5 (text-davinci-003) the day it was released, two days before ChatGPT. Less than four years later, and LLMs are at the center of US politic
For all the focus on hard math, we should also have a bank of historical mysteries and cyphers that we can throw models at. I know a few historians (like https://generativehistory.substack.com/) that
Huh. This number is way off. You’re comparing one human brain to an entire data center capable of training new models. The actual answer is: IQ points per watt: Humans: 5 AI: 7 - 40 (!!) By my rough c
I feel like the data sovereignty movement for firms is going to run headlong into the challenge that maintaining fine-tuned small models in a way that continually beats larger general-purpose models i
I cannot emphasize enough how much GPT-6 Astra and Fable 5.1 are already enough for transformative impact in large sections of the economy. They can reliably do weeks worth of human work when properly
I think once remote connections to “your” Claude or Astra or Claw or whatever become easier and more common, on-phone assistants like Siri will lose a lot of value. You want one smart AI managing ever
Also worth noting the professed anxiety about this, at least in the tone of the posts: Anthropic's post: https://darioamodei.com/post/we-must-pace-the-frontier#top OpenAI's post: https://openai.com/in
As always, statements on X are a mixed bag and I have no secret knowledge to draw on, but if Google is able to return to the frontier on AI, it would definitely change the current AI dynamic, as it is
For context. OpenAI turned to METR for independent investigation of the hugging face incident and Anthropic is inviting them in as a third-party. (Industry self-regulation is not always the best path
This week brought some of the clearest statements we've heard from both Anthropic & OpenAI that some form of recursive self-improvement has been achieved, though it still sounds early RSI will cause r
The strawberry thing was very funny but probably gave people the wrong impression of where AI was heading in math. Every FrontierMath Tier 4 problem has now been solved by AI, with GPT-6 Astra solving
Millennium prize problems are falling, rogue agent swarms are doing heists, lab employees are begging for a slowdown…and the most popular AI videos on YouTube are from cope merchants saying it’s all a
There is a nice visualization of how task bundles change with AI & some good interactive simulations of GDP growth under different assumptions in this. But missing are the right policy responses if we
Another informal nerd benchmark that AIs had struggled with has been passed: Astra successfully designed a Magic the Gathering deck and then used it to defeat a bot on Arena. (the deck may not be stun
Is the @METR_Evals long horizons measure effectively saturated? No updates to the most famous graph of AI progress since May, but other published work suggests you can effectively get 18+ weeks of wor
This is a big one. (And yes, the fights over academic credit and what happened when in the race for the proof need to be resolved, but it is still a big one.) We’re sharing a solution to the Navier-St
Weakly General AI achieved (at least according to the criteria set from 2020). a year before recent predictions ✅Loebner prize was a weak Turing Test, equivalent achieved by GPT-4.5 ✅Winograd passed b
A couple years later, with GPT-6. https://underdark-duel.netlify.app Apparently the mindflayer has a '101-bone skeletal rig' (This is 5E 2014 rules, and I didn't see major errors, though the mind flay
It is less than a decade since the development of the transformer. Less than four years since the release of GPT-3.5 (ChatGPT). Less than two years since the release of o1-preview (the first Reasoner)
It is now getting very hard to track where your work and conversations are in Claude and ChatGPT: Local computer? Cloud? Which computer? Phone? The computer connected via Dispatch or Remote? The one y