Knowledge catalogue
Dwarkesh Podcast: Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements — Breaking down 6 years of pretraining progress
Apple derives a repetition-aware mixture scaling law showing scarce target data can be reused ~15–20× when mixed with generic corpora.
EMO is a pretraining approach that uses mixture of experts (MoE) architecture to develop emergent modularity in language models, enabling different experts to specialize in different types of knowledg
Megatron‑LM PR #7262 introduces a workflow that records ordered per‑rank tensor fingerprints to pinpoint and correct nondeterminism in large‑scale pretraining. By implementing low‑overhead kernel twea
Skild AI: Skild AI unveils S1, a robotics foundation model that it says can learn tasks never seen during pretraining, using a single video demo, without fine-tuning — UNSEEN TASKS10-MINUTE HORIZONSON
Financial Times: Sources: ByteDance is pretraining an AI model with up to 10T parameters, roughly 3x larger than Kimi K3 and larger than the 8T estimate for Anthropic's Mythos 5 — TikTok owner trainin
This article from Latent Space provides guidance on securing employment at cutting-edge AI research laboratories, with a focus on roles related to pretraining large language models. It likely covers p
Anthropic: Anthropic researchers detail “model spec midtraining”, which adds a stage between pretraining and fine-tuning to improve generalization from alignment training — Sara Price2, Samuel Marks2,
Recent advancements in large language models have intensified the need for efficient and deployable models within limited inference budgets. Structured pruning pipelines have shown promise in token ef
COMPASS is a scalable cross‑embodiment robot navigation framework that builds on NVIDIA’s X‑Mobility pretrained policies, adding lightweight RL fine‑tuning specialists for new robot‑scene pairs to avo
The quality of open-weight language models has dramatically improved in recent years. Sharing weights greatly facilitates model adoption by enabling their use across diverse hardware and software plat
Visual generative models (e.g., diffusion models) typically operate in compressed latent spaces to balance training efficiency and sample quality. In parallel, there has been growing interest in lever
Automated lint: 26 errors, 6728 warnings, 3 info
Auto-generated index of all papers mentioned across the wiki.
Auto-generated index of all models mentioned across the wiki.
Auto-generated index of all tools mentioned across the wiki.
Auto-generated index of all companies mentioned across the wiki.
Auto-generated index of all people mentioned across the wiki.
Multilingual self-supervised speech models can benefit from sharing information across languages, but under a matched total pretraining data budget they still fall short of monolingual models. We show
An Anthropic PBC researcher has resigned over concerns that artificial intelligence labs are “gambling with our lives.” Jacob Coxon was part of the company’s AI pretraining team until today. He announ
Automated lint: 12 errors, 53990 warnings, 3 info
Standard Bots develops AI‑native industrial robot arms (used for machine tending, welding, assembly) that rely on a shared pretrained base model with 1–2 billion parameters. The model is adapted to ea
Automated lint: 14 errors, 54840 warnings, 3 info
Automated lint: 12 errors, 54839 warnings, 3 info
It is impossible to make large language models fully secure against hacks because of a fundamental flaw in how they work, a team of researchers argue in a paper presented at the International Conferen
At Google Cloud Next ’26 we announced Cloud Storage Rapid, a family of object storage capabilities for data-intensive workloads like AI and analytics. Out of the gate, Cloud Storage Rapid consists of
This article outlines Sebastian Raschka's systematic approach to learning and understanding Large Language Model (LLM) architectures, likely covering foundational concepts, key components, and methodo
NVIDIA’s Seattle Robotics Lab and Isaac engineering team created reusable robotic systems to automate GB300 tester‑tray assembly, achieving over 95 % success on busbar tasks with a 160‑second cycle ti
Automated lint: 19 errors, 64915 warnings, 3 info
Auto-generated synthesis of 18054 entries about arxiv-cs-cv
Automated lint: 17 errors, 59669 warnings, 3 info
NVIDIA Cluster Readiness Engine (NVCRE) is an open‑source Kubernetes controller that runs real distributed workloads across topology‑aware node groups to certify GPU cluster readiness before productio
Discrete flow matching generates text by iteratively transforming noise tokens into coherent language, but may require hundreds of forward passes. Distillation uses the multi-step trajectory to train
The Guardian: Matt Clifford steps down as the chair of the UK government's science and tech research unit after joining Anthropic, following conflict of interest concerns — Matt Clifford forced to sta
Michael Nuñez / VentureBeat: The Agentic AI Foundation updates MCP with a fully stateless architecture, a hardened authentication model, a formal 12-month deprecation policy, and more — The Model Cont
How we built the OlmoEarth Platform to fine-tune geospatial models and run continent-scale satellite inference while managing massive data pipelines, distributed compute, and automatically recovering
Automated lint: 20 errors, 8743 warnings, 3 info
Ornith-1.0: Self-Scaffolding LLMs for Agentic Coding This is an interesting new open weights (MIT licensed) model, the first model release from DeepReinforce. [...] with variants including 9B Dense, 3
If Claude Fable stops helping you, you'll never know Jonathon Ready highlights one of the more eyebrow-raising details from the 319 page system card for Fable 5 and Mythos 5. Here's a longer excerpt,
At Google Cloud Next, we are announcing innovations across every layer of our storage stacks — performance, intelligence, and management — to ensure your data is as fast and as useful as the AI models
This paper was accepted at the Workshop on Navigating and Addressing Data Problems for Foundation Models (NADPFM) at ICLR 2026. Principled domain reweighting can substantially improve sample efficienc
This paper was accepted at the Workshop on Memory for LLM-Based Agentic Systems at ICLR. Language models have consistently grown to compress more world knowledge into their parameters, but the knowled
Muse Spark is the first model from Meta's Superintelligence Labs division and the debut entry in the new Muse model family, representing a 'ground-up overhaul' of the company's AI efforts. It is ...