Knowledge catalogue
Can an LM, starting from random init (!!), learn to generate all of its pretraining data? Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator pro
Excited to share that I’ve joined OpenAI in London to work on pretraining! I’ve spent the last few years on pretraining and long-context, and I’ll be helping grow the London team. We’re hiring excepti
Cool paper from Meta FAIR. It's on self-improving LLMs but on the pretraining side. (bookmark it) Most LLM safety, factuality, and reasoning fixes get bolted on at post-training. By then, the patterns
ERNIE 5.1 is here 🚀 ERNIE 5.1 significantly reduces pretraining cost while compressing total parameters to ~1/3 and activated parameters to ~1/2 — using only ~6% of the pretraining cost compared to mo
Oops, SIGReg did it again! Large scale (CC12M->Datacomp-L) vision-language JEPA pretraining beats CLIP and SigLIP objectives! Thanks to SIGReg, our LeVLJEPA has no collapse, no EMA, no stop-gradient,
🔥 We introduce LeVLJEPA: the first fully non-contrastive end-to-end vision-language pretraining method competitive with CLIP & SigLIP 💪🏼 👀 No negatives. No temperature. No momentum encoder. No teacher
What we find most useful about this study is the decomposition. For byte-level pretraining, the ordering is throughput first and boundary signal second; the boundary benefit can be recovered without a
Today we release a study on decoupling the benefits of subword tokenization for language model training, by simulating each suspected benefit one at a time inside a 1.7B byte-level pretraining pipelin
Today we release Token Superposition Training (TST), a modification to the standard LLM pretraining loop that produces a 2-3× wall-clock speedup at matched FLOPs without changing the model architectur
What we find most useful about TST is the decoupling. The training-time efficiency is fully separated from the inference-time architecture, which makes TST a clean addition on top of other pretraining
Cool idea from Nous Research. What if you could speed up long-context pretraining with a subquadratic wrapper that you remove before deployment? That is the idea behind Lighthouse Attention. The metho
i like that there are models called bert and ernie, but in all seriousness, this update looks impressive ERNIE 5.1 is here 🚀 ERNIE 5.1 significantly reduces pretraining cost while compressing total pa
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving
A significant fraction of humanity read this thread. Is anyone prepared to do something about it? Fretting doesn’t count. I resigned from Anthropic today. I spent the last three years doing pretrainin
Anyone still denying the risks posed by unregulated AI should read this thread. It’s time to pump the brakes & take action NOW. We need: - Mandatory safeguards - A comprehensive testing regime - Urgen
I dropped out of a PhD in AI safety partially for the opposite reason. I didn’t believe AI existential risk was as important as the doomers think. My biggest pet peeve is that no one can really provid
I worked at Google DeepMind and now at Anthropic. This is a common sentiment amongst my peers. (I write this in personal capacity.) There is not yet a viable scientific plan to solve risks from recurs
read the whole thread. although i don’t 100% agree with certain bits it is compelling and informed from the inside perspective that definitely deserves to be heard. I resigned from Anthropic today. I
Why are AI employees willing to gamble with our lives? Many AI employees hold a misanthropic belief: a cosmos of blissful AIs is worth risking human extinction. Utilitarians shouldn't decide the fate
This is for pretraining based on nemotron nvp4 figures, mfu etc Large context, multimodal, long RL we are seeing now could make it a multiple of this Worth noting how good the lite model is, similar t
It’ll be interesting to see if the post training for this uses a multiple of the compute of pretraining as cursor did when they tuned Kimi as the base model Grok foundation model V9-Medium (1.5T) has
We've gone even farther: Nemotron 3 Super is 120B and pretrained on 25T tokens in NVFP4. Nemotron 3 Ultra is ~500B and also pretrained in NVFP4. Accelerated computing means we rethink every aspect of
verbalizing one of those aha moments i had that seems retroactively pretty obvious: if you prioritize pretrain data quality enough that commoncrawl isn't good enough for you, you have to build a Whole
Introducing nanowhale 🐳! A tiny DeepSeek model fully pretrained by an agent. Inspired by @karpathy's nanochat, we gave ml-intern the task of training a tiny MoE with all the architectural advancements
Pretrained ViTs see the world in rich, dense detail. Most policies pool it to a single vector before acting, discarding most of it. We introduce Patch Policy: a minimal architectural extension that en
This was right five years ago, and still is: “Large scale pretrained models are certainly likely to figure prominently in artificial intelligence for the near future, and play an important role in com
which is NOT new; see this quote from 5 years ago. the fact that all this is still true says a lot. This was right five years ago, and still is: “Large scale pretrained models are certainly likely to
To train better open models, we need predictable scaling. Delphi is Marin’s first step: we pretrained many small models with one recipe, then extrapolated 300× to predict a 25B-param / 600B-token run
Sapiens2 is the highest quality ViT backbone that now exists in the public domain. It was pretrained on the equivalent of 1/2 of all human images on Flickr. First public release by a large lab that is
🦤 LeWorldModel: Learning Physics from Pixels — Stable World Models with Just Two Losses World models: 1️⃣ DINO-WM: pretrained ViT encoder (from ImageNet) → features → predictor. But encoder is frozen,
rsi is here. jesus https://x.com/nickevanjoseph/status/2056760504949842219?s=46 Excited to welcome Andrej to the Pretraining team! He'll be building a team focused on using Claude to accelerate pretra
Next year frontier models are going to one shot everything 100x faster & cheaper than they are No reasoning traces to monitor Instinctive intelligence Which is why what you embed in pretraining is eve
Impressive new paper from Meta. (bookmark it) Scaling laws assume model size and training data act on loss independently. This work introduces Skaling law, which couples capacity and data through a si
Great to see this out, proud of the team!!! @AMD Instella 16B MoE foundation model, fully open with checkpoint after each stage from pretraining, long-context, SFT, DPO and RL. Dataset detail, trainin
Learning causality from internet videos in latent space first, and then using RL to teach the foundation model how to act. This approach is 30× cheaper than Gemini 3.1 Flash on pretraining and achieve
As believers of open research, we are disappointed to see Anthropic silently degrading Fable 5 for AI development 'Any topic related to building pretraining pipelines, distributed training infrastruct
The Top AI Papers of the Week (May 4 - 10) - Conductor - HeavySkill - Horizon Generalization - 1,000 Synthetic Computers - Self-Improving Pretraining - Coordination as Architecture - Connect Four Alph
Technical highlights: CogViT Vision Encoder - Built with dual-teacher distillation: SigLIP2 for semantics, DINOv3 for texture. A two-stage recipe, masked modeling, then contrastive pretraining, with Q
6 billion tokens for $65 Is it as good as Astra and Fable at hard stuff? No of course not. But you don't always need a genius and can just call one for advice when you do. Extrapolate where this is go
NEW WORLD MODEL: @ylecun's team is back with an efficient model! This project involves @ylecun, @lukaskuhn77, @lucasmaes_, @quentinlldc, and @randall_balestr. A couple definitions first: - DINO: self-
The sense of touch is the most criminally under-explored modality in robotics. Imagine doing sleight of hand wearing thick oven mitts. That's exactly how a robot feels today if it were alive. A magnet
Four small architecture decisions can cost up to 47% of a model's long-context performance. New research from Ai2, Carnegie Mellon, and the University of Washington isolates them. Normalization, GQA,
World Modeling with JEPA has recently gained traction thanks to a novel anti-collapse mechanism called 'SIGReg' (by @ylecun and @randall_balestr). The math is clean, but it is rarely explained from fi
congrats on the launch! As the task horizons on agents gets longer, we need more work evaluating and training models to be better in long-running, open-ended, evolving real-world environments. Today w
VLA-JEPA just dropped in LeRobot 🤖 What makes this model special is that it does not just learn what action to take from a given observation, it also leverages a JEPA world model to learn action-relev
1/ today we're releasing muse spark, the first model from MSL. nine months ago we rebuilt our ai stack from scratch. new infrastructure, new architecture, new data pipelines. muse spark is the result
Great report from Google and MIT. Co-designing ML models and the accelerators is an exciting research direction. This report introduces Coco, an agent platform TPU architects use for hardware and mode
Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO). 00:00 Introduction
Reasoning from scratch, round number 5! This time, talking about log-probability scoring (also a great fundamental concept for loss functions like cross-entropy in pre-training and distillation) and s
The current AI and I suspect that all AI of the near future (few human generations that is) is in fact a more sophisticated form of data access methods in the same category as classic database indexes
Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent i
Looped transformers are a popular architecture topic right now. This new technical report extends the loop across tokens. Recurrent Looped Transformer (RLT) makes the decoder recurrent over every toke
The Coxon psyop was a decade and more than a billion dollars in the making. Here's the story. Coefficient Giving (formerly Open Philanthropy) is the grant-providing vehicle of Dustin Moskowitz. CG is
“We can’t slow down because China will beat us.” We can slow down. Chinese AI is waterskiing behind the US. Their AI capabilities come at a delay since they mainly come from distilling US AIs. If we s
Anthropic: Most hypocritical company in history? 👉 Screams about how AI will kill everyone, while racing ahead. 👉 Moans about distillation while distilling the world’s books, articles, and other writi
// human-values.pseudo | ARTIFACT 0.1.0-experimental.1 | BASE 1.2.0-rc1 | EDITION ai-compact | NARRATOR System_Synthesis_Node_01 | PROVENANCE human-directed, AI-assisted | STATUS single-encoder resear
Jacob was a senior researcher who joined Anthropic in May. My Anthropic colleagues and I had been trying to recruit him for ~2 years, because we knew he was a strong researcher at OpenAI. Before he le
Reasoning from scratch round 2: In this video, I cover the text generation process in LLMs and KV caching (to prepare the base model before adding reasoning techniques in the upcoming ones). 00:00 Int
Banger paper from Microsoft and Cornell. If you have looked at thinking tokens and decided you cannot afford the context, read this one. (bookmark it) A pause token buys the model extra compute for ea
Introducing Z1T: Our first family of transformer-like models made for sparse probabilistic hardware like Z1 Achieving up to 140x energy efficiency gains over GPUs and revealing a new scaling law for s