Knowledge catalogue
Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days. Their Meta Muse Glimmer model is a 30B multimodal reasoning model with a Gemma-like a
Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub! This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people. Thanks
Consistent with what I found with Qwen3.6 a while back: Claude Code uses 2-3x as many tokens than (many) other harnesses at similar success rate. - Unoptimized? - Buggy? - Deliberate (coz that helps i
Just for reference, from what I observed in my Using Local Coding Agents blog article last month: https://x.com/rasbt/status/2070518167399698490?s=20 'I tried to analyze why Claude Code uses more toke
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up prod
Looking forward to chatting with @hugobowne today (July 27) at 4 pm PT on the Vanishing Gradient livestream on YouTube. Will cover open source, the newest LLMs & trends, agent frameworks, and whatever
Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up outside the closed labs. Plus, it gives us the freedom to run
Interesting surprise drop from Thinky! The Inkling model looks pretty solid on benchmarks, and it has some little surprises in its architecture: - Small conv layers in several places - An RMSNorm for
This post addresses a question comparing 'Ultra' and 'Pro' product tiers or model versions, likely clarifying whether they represent meaningfully different offerings or are simply alternative names fo
For agentic coding, one can say: - Unless you need Terra Ultra perf, it's always better to use a Luna model with higher effort setting (same or better performance but cheaper). - Forget everything bel
This post discusses the conceptual distinction between training-scaling and inference-scaling in machine learning models, while acknowledging related frameworks or concepts (possibly referencing Sol,
I like choices... but now I have: 2x modes (Codex vs. Work mode) 3x GPT-5.6 models (Sol, Terra, Luna) 5x effort levels (Light, Medium, High, Extra High, Ultra) That's 2 x 3 x 5 = 30 possible configura
After 18 months of writing, coding, and experimenting, Build a Reasoning Model (From Scratch) is finally out! My first copies just arrived! 📚 440 full-color pages. Inference scaling, reinforcement lea
Sebastian Raschka announced the availability of a book through Manning Publications (with immediate shipping) and Amazon (available for preorder with delivery in a few weeks). The specific book title
I put together a new article on setting up local coding agents with open-weight models. Everything runs 100% locally. I thought it might be useful putting this together because many people asked me ab
This article explores the implementation and use of local coding agents—AI systems designed to autonomously write, test, and debug code on local machines without relying on external APIs. The piece li
Have been taking different local open-weight LLMs for a test drive in different harnesses (Qwen-Code, Codex, Claude Code). 30B Mixture-of-Expert models are kind of a nice sweet spot and can solve chal
This post traces the historical lineage of machine learning techniques, suggesting that LatentMoE builds upon a chain of progressively developed methods including MLA, LoRA, and SVD, emphasizing how m
Turns out Fable 5 is shadowbanning AI researchers 🫤 mythos will be bad ON PURPOSE on ai 'frontier llm research' tasks, this is very very sad for the research community also the fact that this is un pu
And another open-weight release. Nemotron 3 Ultra has an ultra impressive capability:efficiency ratio! Design-wise, it carries forward the Mamba-2-attention hybrid stack and LatentMoE introduced in th
This post highlights four recent improvements to the ecosystem of open-weight large language models designed to run efficiently on consumer hardware, covering developments that make local LLM deployme
The MiniMax M2 series was one of the most widely used open-weight LLM series earlier this year. Now, we got a technical report with some interesting tidbits. I summarized some of them below: 1. Full a
Added a DeepSeek Sparse Attention (DSA) from-scratch implementation to my LLMs-from-scratch repo thanks to an awesome new reader contrib. With motivation, overview, and GPT-style model reference imple
Gated DeltaNet has been one of my favorite 'hybrid attention' newcomers in the good old transformer stack. Excited to see Gated DeltaNet-2. Adding it to my reading stack. In the meantime, I have a pri
It's been *almost* a bit quiet around LLM architecture releases in the past two weeks 😅 Interesting tidbit is the parallel block design. Via the Cmd-A the tech report 'equivalent performance but signi
New article: a visual tour of recent LLM architecture advances, from Gemma 4 to DeepSeek V4. I focus on long-context efficiency tweaks like KV sharing, per-layer embeddings, layer-wise attention budge
DeepSeek maintains the highest efficiency in terms of active parameters relative to total model size, outperforming competitors in the ratio of parameters actually used during inference versus total t
Sebastian Raschka shared an HTML-formatted table comparing active parameter ratios across different large language model architectures for improved readability and to avoid text truncation. The resour
A little talk on what we can learn from implementing LLM architectures from scratch in Python and PyTorch. And how I approach new open-weight models, compare them against reference implementations etc
Interesting paper. What I like about this is that it is a relatively low-commitment attention modification. I.e., one can use it during most of training, switch back to vanilla attention near the end,
Back from a little family break! Lots has happened, and I’m planning to do a deeper dive into the most interesting architectural components (soon). Btw, are there any major architectures I missed belo
Sebastian Raschka maintains a comprehensive gallery of LLM architecture diagrams and visualizations on his website, with higher-resolution versions available at sebastianraschka.com/llm-architecture-g
This post documents a second wave of large language model and AI architecture releases from April, featuring updates from multiple organizations including Ant Ling 2.6 1T, Minimax M2.7, Xiaomi MiMo V2
April saw significant activity in large language model releases, with five major models introduced including Gemma 4, GLM-5.1, Qwen 3.6, Kimi K2.6, and DeepSeek V4. These releases have been added to a
Sebastian Raschka has updated his LLM architecture gallery with higher resolution figures and improved summaries, including coverage of the DeepSeek V4 model architecture. This resource provides visua
Had a great time at PyCon & PyData DE. Highly recommend it. Great open-source, community-focused conference with lots of builders in the Python AI, LLM and agent space. Taking a short family break, my
I’ve uploaded a new paper on arXiv (co-authored by @rasbt): MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning In Parameter-Efficient Fine-Tuning, a key question may not just be how low-rank th
Sebastian Raschka (rasbt) used Randal Olson's Tufte Test tool, developed by Goodeye Labs, to improve the visual quality of a machine learning benchmark plot. The Tufte Test encodes seven of Tufte'...
Strong release! GLM-5.1 is a DeepSeek-V3.2-like architecture (including MLA and DeepSeek Sparse Attention) but with more layers. And the benchmarks look better throughout! Looks like THE flagship open