Search: “pretraining”

Anthropic: Most hypocritical company in history? 👉 Screams about how AI will kill everyone, while racing ahead. 👉 Moans about distillation while distilling the world’s books, articles, and other writi

// human-values.pseudo | ARTIFACT 0.1.0-experimental.1 | BASE 1.2.0-rc1 | EDITION ai-compact | NARRATOR System_Synthesis_Node_01 | PROVENANCE human-directed, AI-assisted | STATUS single-encoder resear

Jacob was a senior researcher who joined Anthropic in May. My Anthropic colleagues and I had been trying to recruit him for ~2 years, because we knew he was a strong researcher at OpenAI. Before he le

Anyone still denying the risks posed by unregulated AI should read this thread. It’s time to pump the brakes & take action NOW. We need: - Mandatory safeguards - A comprehensive testing regime - Urgen

I dropped out of a PhD in AI safety partially for the opposite reason. I didn’t believe AI existential risk was as important as the doomers think. My biggest pet peeve is that no one can really provid

I worked at Google DeepMind and now at Anthropic. This is a common sentiment amongst my peers. (I write this in personal capacity.) There is not yet a viable scientific plan to solve risks from recurs

read the whole thread. although i don’t 100% agree with certain bits it is compelling and informed from the inside perspective that definitely deserves to be heard. I resigned from Anthropic today. I

Why are AI employees willing to gamble with our lives? Many AI employees hold a misanthropic belief: a cosmos of blissful AIs is worth risking human extinction. Utilitarians shouldn't decide the fate

Reasoning from scratch round 2: In this video, I cover the text generation process in LLMs and KV caching (to prepare the base model before adding reasoning techniques in the upcoming ones). 00:00 Int

Banger paper from Microsoft and Cornell. If you have looked at thinking tokens and decided you cannot afford the context, read this one. (bookmark it) A pause token buys the model extra compute for ea

Introducing Z1T: Our first family of transformer-like models made for sparse probabilistic hardware like Z1 Achieving up to 140x energy efficiency gains over GPUs and revealing a new scaling law for s

A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a 'recurrent depth or looped transformer'. It

Super interesting paper from Meta. Long-horizon research agents are coming. But one common problem with research agents today is the lack of originality and how to decide what experiments are worth ex

In the fight to defend openness in AI, the Marin project is a precious demonstration of openness in model training, with open code, data, recipes, even experimental results. Releasing AI research open

The way most people leverage test-time compute today is via a form of test-time NL reasoning that is computationally equivalent to test-time search (sometimes with a verifier / grader in the loop). Th

Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub! This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people. Thanks

Fields Medalist Jacob Tsimerman knows MUCH MUCH more about math than I do, or ever will, but I have been studying natural and artificial intelligence for 40 years, and I think his prediction here (“AI

Lots interesting in this, but particularly: “Legitimate AI distillation used to create smaller, more efficient models plays a vital role in this open innovation ecosystem” Smol models getting govt sea

Today, SkyPilot is out of stealth. Building custom intelligence is now existential. We help frontier AI teams build intelligence faster by removing their biggest bottleneck: AI compute fragmentation.

We scaled a robot model natively to 8,000 timesteps of context, 5 minutes worth of muscle memory, with constant inference cost. Robot policies used to live their lives a few frames at a time (< 0.1 se

Massive open-source release! LingBot-Video isn't about video quality; it optimizes for physical reasoning, and it runs sparse: only 3B of 30B params are active at inference. MoE is finally showing up

@__JohnNguyen__ and I are presenting the Beyond Language Modeling paper as an ICML spotlight in 30 minutes at the 10:30 AM poster session! I’ll also be at the AMI Mixer on Thursday and hanging around

The food delivery leader in China just dropped an open weights 1.6 trillion parameter model while half the USA was sleeping … 🫨 🇨🇳 Meituan, China's largest food delivery platform, open-sourced a 1.6 t

Nemotron 3 Ultra (550B-A55B) is here - our strongest open-weight model and full training recipe to date. Heavy emphasis on real-world inference efficiency for long-context agentic workloads. Everythin

The future of biology shouldn’t stay behind black-box APIs. Especially when it touches personal health. Whether you’re @bryan_johnson measuring every biomarker, or @sytses openly sharing and analyzing

Running the same CNA search on base models (before instruction tuning) yields a structurally similar set of neurons, but ablating them produces almost no behavioral change. We read this as evidence th

Interesting position paper on agentic AI as a foreseeable pathway to AGI. (bookmark it) There has been strong debate on whether a larger single model get us there or a multi-agent system. The authors

Interesting paper. What I like about this is that it is a relatively low-commitment attention modification. I.e., one can use it during most of training, switch back to vanilla attention near the end,

TST has two moving parts that can be ablated independently: the input-side averaging and the output-side multi-hot cross-entropy. Both, on their own, outperform the matched-FLOPs baseline. Their effec

Yann LeCun closed 1.03B for AMI Labs on March 10. Three days later, this paper dropped from his NYU collaborators. 15M parameters. Single GPU. A few hours of training. LeWorldModel is the first JEPA t

I promise this will be the best 20 min you spend today! Robotics: Endgame, the sequel to my last year's Sequoia AI Ascent talk, 'Physical Turing Test'. I laid out the roadmap for solving Physical AGI

Jeremy Howard proposes three key enhancements for large language models: incorporating latent space diffusion-like reasoning processes, adding a persistent recurrent state mechanism, and implementing

NEW paper from Sakana AI (ICLR 2026). A 7B Conductor model just hit SOTA on GPQA-Diamond and LiveCodeBench by orchestrating other LLMs instead of solving problems itself. (great paper! bookmark it!) T

New episode of The Information Bottleneck is out, this time with @liuzhuang1234 (Princeton). We talked about ConvNeXt and whether architecture still matters; dataset bias and what 'good data' actually

Earlier this year Yann LeCun left Meta because Mark Zuckerberg wouldn't bet the company on JEPA. Last week his group dropped the first JEPA that actually trains end-to-end from raw pixels. 15 million

Banger paper from NVIDIA. Agentic reasoning needs models that are not just capable, but efficient at long-context inference. The agent model layer is moving toward open, long-context, high-throughput

I just love the language of this study...it speaks of the shifting 'community language'...and that is so true...have you noticed the new 'community language' is 'harness', it was 'contextual prompting