New episode of The Information Bottleneck is out, this time with @liuzhuang1234 (Princeton). We talked about ConvNeXt and whether architectu…

New episode of The Information Bottleneck is out, this time with @liuzhuang1234 (Princeton). We talked about ConvNeXt and whether architecture still matters; dataset bias and what "good data" actually looks like; ImageBind and why vision is the natural bridge across modalities; CLIP's blind spots; memory as the real bottleneck behind the agent hype; whether LLMs have world models; and Transformers Without Normalization. For years, the vision community debated what actually matters: architecture, inductive bias, self-attention vs convolution. After a lot of back-and-forth, we ended up in a funny place: ViT and ConvNet give roughly the same performance once you tune the details. What I find interesting is that once you reach a certain performance level, it becomes much easier to swap and tweak components without really changing the outcome. Talking to Zhuang on this episode, I kept wondering whether the same is now true for LLMs. If we wil spent serious time on an alternative architecture today, would you actually get a meaningfully different model, or just land on the same Pareto curve with extra steps? I'm starting to suspect it's the latter. Architecture matters less than we think. Data, compute, and a handful of pillars do most of the work. Media

Related

• 🦤 LeWorldModel: Learning Physics from Pixels — Stable World Models with Just Two Losses World models: 1️⃣ DINO-WM: pretrained ViT encoder (…
• Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution
• Counting Worlds Branching Time Semantics for post-hoc Bias Mitigation in generative AI
• EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World

Source: Yann LeCun (X) | 2026-04-24