Search: “sebastian-raschka”

Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO). 00:00 Introduction

During a Swyx podcast interview in October 2026, an OpenAI engineer revealed that the company’s Decision API is built on the GPT‑6 Luna model, augmented with a dedicated decision head. This configurat

Took a while, but here it is, a mega write-up on language models for text classification and, of course, Jev. It's basically a visual guide to RNNs, CNNs, transformers, and calibration, with hands-on

A little Ember-1 tl;dr. Seems like a great model! (Was recently asked on a podcast, given $ xx million, what's the best way to develop a frontier LLM today? My recommendation was: start with an existi

Also kudos to fireworks AI for admitting that they built on top of Kimi (I remember the awkward Mistral 3 release) https://x.com/rasbt/status/1999557830131494977?s=20 Hold on a sec, Mistral 3 Large us

Reasoning from scratch, round number 5! This time, talking about log-probability scoring (also a great fundamental concept for loss functions like cross-entropy in pre-training and distillation) and s

MiMo-V2.6 is 'simply' the best (for now). Despite its simple architecture design it's currently No.1 in the open-weight benchmarks (weighted average). With 'simple,' I mean a classic Grouped Query Att

Inference scaling part 1. Starting with a modded text generation function (temperature scaling, top-p filtering, multinomial sampling) to generate diverse outputs for self-consistency and best-of-N (i

Some food for thought when designing benchmarks... So, here's a little computer-use (visual) comparison between GPT-5.6 Astra and Qwen3.8 Max. The task here was to recreate the image in the center usi

Reasoning from scratch round 3: This time, I cover generating a verifier for... a) ...evaluation (base model versus any future model improvement) b) ...the reinforcement learning with verifiable rewar

Big overhaul on DeepSeek V4.1 using an encoder-decoder setup. Tbh they should have called it DeepSeek V5! Super cool and refreshing, though! 🚀 Introducing DeepSeek-V4.1-Flash: smarter, faster, more ef

Nice showcase that interesting LLM work can be done on single GPU! I extended the GPT-2-style code from @rasbt's 'Build a Large Language Model (from Scratch)' so that it was a 6-expert (2 active) mixt

Sebastian Raschka’s September 2026 article introduces **GPT‑6 *Astra***, a next‑generation language model that expands upon the Transformer backbone with a novel *looped* architecture—allowing multipl

I put together a mega write-up on GPT-6 Astra & looped transformers. How looped transformers / recurrent depth works, cost-tradeoffs, whether it hides reasoning traces, with lots of figures and a tour

Reasoning from scratch round 2: In this video, I cover the text generation process in LLMs and KV caching (to prepare the base model before adding reasoning techniques in the upcoming ones). 00:00 Int

A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a 'recurrent depth or looped transformer'. It

Now we know: The popular Ox Alpha LLM was GLM-5.3-Flash... Compared to GLM-5.2, this new GLM-5.3-Flash model uses: - a Kimi Linear-style 3:1 (super*) hybrid attention pattern with 34 Kimi Delta Attent

A couple of days ago, I did a quick explainer on Claude’s new watermarking process and implementation. Since it’s such a popular topic and sparked such a lively discussion, I thought it might be inter

Nice case study on using optimized functions whenever possible (except for educational purposes, though 😆) By switching from a hand-rolled GELU to PyTorch's built-in one, I improved my LLM training sp

A short illustration of how the Claude's watermarking is supposed to work (based on my read of their released materials). In general, when we are generating tokens, there can be multiple high-scoring

8/19, join us for the #ACMTechTalk, 'From Conventional LLMs to Reasoning Models to Agents,' w/AI & LLM Research Engineer @rasbt. ACM Practitioner Board Co-Chaior @marlene_zw (@Microsoft) will moderate

Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days. Their Meta Muse Glimmer model is a 30B multimodal reasoning model with a Gemma-like a

Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub! This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people. Thanks

Consistent with what I found with Qwen3.6 a while back: Claude Code uses 2-3x as many tokens than (many) other harnesses at similar success rate. - Unoptimized? - Buggy? - Deliberate (coz that helps i

Just for reference, from what I observed in my Using Local Coding Agents blog article last month: https://x.com/rasbt/status/2070518167399698490?s=20 'I tried to analyze why Claude Code uses more toke

The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up prod

Looking forward to chatting with @hugobowne today (July 27) at 4 pm PT on the Vanishing Gradient livestream on YouTube. Will cover open source, the newest LLMs & trends, agent frameworks, and whatever

Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up outside the closed labs. Plus, it gives us the freedom to run

Interesting surprise drop from Thinky! The Inkling model looks pretty solid on benchmarks, and it has some little surprises in its architecture: - Small conv layers in several places - An RMSNorm for

This post addresses a question comparing 'Ultra' and 'Pro' product tiers or model versions, likely clarifying whether they represent meaningfully different offerings or are simply alternative names fo

For agentic coding, one can say: - Unless you need Terra Ultra perf, it's always better to use a Luna model with higher effort setting (same or better performance but cheaper). - Forget everything bel

This post discusses the conceptual distinction between training-scaling and inference-scaling in machine learning models, while acknowledging related frameworks or concepts (possibly referencing Sol,

I like choices... but now I have: 2x modes (Codex vs. Work mode) 3x GPT-5.6 models (Sol, Terra, Luna) 5x effort levels (Light, Medium, High, Extra High, Ultra) That's 2 x 3 x 5 = 30 possible configura

After 18 months of writing, coding, and experimenting, Build a Reasoning Model (From Scratch) is finally out! My first copies just arrived! 📚 440 full-color pages. Inference scaling, reinforcement lea

Sebastian Raschka announced the availability of a book through Manning Publications (with immediate shipping) and Amazon (available for preorder with delivery in a few weeks). The specific book title

I put together a new article on setting up local coding agents with open-weight models. Everything runs 100% locally. I thought it might be useful putting this together because many people asked me ab

This article explores the implementation and use of local coding agents—AI systems designed to autonomously write, test, and debug code on local machines without relying on external APIs. The piece li

Have been taking different local open-weight LLMs for a test drive in different harnesses (Qwen-Code, Codex, Claude Code). 30B Mixture-of-Expert models are kind of a nice sweet spot and can solve chal

This post traces the historical lineage of machine learning techniques, suggesting that LatentMoE builds upon a chain of progressively developed methods including MLA, LoRA, and SVD, emphasizing how m

Turns out Fable 5 is shadowbanning AI researchers 🫤 mythos will be bad ON PURPOSE on ai 'frontier llm research' tasks, this is very very sad for the research community also the fact that this is un pu

And another open-weight release. Nemotron 3 Ultra has an ultra impressive capability:efficiency ratio! Design-wise, it carries forward the Mamba-2-attention hybrid stack and LatentMoE introduced in th

This post highlights four recent improvements to the ecosystem of open-weight large language models designed to run efficiently on consumer hardware, covering developments that make local LLM deployme

The MiniMax M2 series was one of the most widely used open-weight LLM series earlier this year. Now, we got a technical report with some interesting tidbits. I summarized some of them below: 1. Full a

Added a DeepSeek Sparse Attention (DSA) from-scratch implementation to my LLMs-from-scratch repo thanks to an awesome new reader contrib. With motivation, overview, and GPT-style model reference imple

Gated DeltaNet has been one of my favorite 'hybrid attention' newcomers in the good old transformer stack. Excited to see Gated DeltaNet-2. Adding it to my reading stack. In the meantime, I have a pri

It's been *almost* a bit quiet around LLM architecture releases in the past two weeks 😅 Interesting tidbit is the parallel block design. Via the Cmd-A the tech report 'equivalent performance but signi