TechniqueRLHF / Alignment2 recent entries15 Aug 2026Building an AI Text Detector From ScratchThe article describes an end‑to‑end DIY project that builds an AI text detector from scratch, including dataset construction, model training, local deployment, and a reinforcement learning verifier (R→3 Oct 2026Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Gr…Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO). 00:00 Introduction
TechniqueRAG1 recent entries19 Sept 2026Inference scaling part 1. Starting with a modded text generation function (temperature scaling, top-p filtering, multinomial sampling) to ge…Inference scaling part 1. Starting with a modded text generation function (temperature scaling, top-p filtering, multinomial sampling) to generate diverse outputs for self-consistency and best-of-N (i
TechniqueAgents8 recent entries26 Jul 2026Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up o…Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up outside the closed labs. Plus, it gives us the freedom to run→27 Jul 2026Looking forward to chatting with @hugobowne today (July 27) at 4 pm PT on the Vanishing Gradient livestream on YouTube. Will cover open sour…Looking forward to chatting with @hugobowne today (July 27) at 4 pm PT on the Vanishing Gradient livestream on YouTube. Will cover open source, the newest LLMs & trends, agent frameworks, and whatever→30 Jul 2026Just for reference, from what I observed in my Using Local Coding Agents blog article last month: https://x.com/rasbt/status/207051816739969…Just for reference, from what I observed in my Using Local Coding Agents blog article last month: https://x.com/rasbt/status/2070518167399698490?s=20 'I tried to analyze why Claude Code uses more toke→30 Jul 2026Consistent with what I found with Qwen3.6 a while back: Claude Code uses 2-3x as many tokens than (many) other harnesses at similar success …Consistent with what I found with Qwen3.6 a while back: Claude Code uses 2-3x as many tokens than (many) other harnesses at similar success rate. - Unoptimized? - Buggy? - Deliberate (coz that helps i→11 Aug 2026Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days. Their Meta Muse Glimmer m…Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days. Their Meta Muse Glimmer model is a 30B multimodal reasoning model with a Gemma-like a→14 Aug 20268/19, join us for the #ACMTechTalk, 'From Conventional LLMs to Reasoning Models to Agents,' w/AI & LLM Research Engineer @rasbt. ACM Practit…8/19, join us for the #ACMTechTalk, 'From Conventional LLMs to Reasoning Models to Agents,' w/AI & LLM Research Engineer @rasbt. ACM Practitioner Board Co-Chaior @marlene_zw (@Microsoft) will moderate→22 Sept 2026MiMo-V2.6 is 'simply' the best (for now). Despite its simple architecture design it's currently No.1 in the open-weight benchmarks (weighted…MiMo-V2.6 is 'simply' the best (for now). Despite its simple architecture design it's currently No.1 in the open-weight benchmarks (weighted average). With 'simple,' I mean a classic Grouped Query Att→27 Sept 2026Also kudos to fireworks AI for admitting that they built on top of Kimi (I remember the awkward Mistral 3 release) https://x.com/rasbt/statu…Also kudos to fireworks AI for admitting that they built on top of Kimi (I remember the awkward Mistral 3 release) https://x.com/rasbt/status/1999557830131494977?s=20 Hold on a sec, Mistral 3 Large us
TechniqueFine-tuning7 recent entries8 Apr 2026PS: I finally got around to trying out @randal_olson 's Tufte Test tool to prettify the benchmark plot. Great tool 👌! https://www.goodeyela…Sebastian Raschka (rasbt) used Randal Olson's Tufte Test tool, developed by Goodeye Labs, to improve the visual quality of a machine learning benchmark plot. The Tufte Test encodes seven of Tufte'...→8 Apr 2026I’ve uploaded a new paper on arXiv (co-authored by @rasbt): MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning In Parameter-Efficient…I’ve uploaded a new paper on arXiv (co-authored by @rasbt): MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning In Parameter-Efficient Fine-Tuning, a key question may not just be how low-rank th→9 Jun 2026Always back to the basics: LatentMoE was probably inspired by MLA, which was inspired by LoRA, which was inspired by SVD, which was inspired…This post traces the historical lineage of machine learning techniques, suggesting that LatentMoE builds upon a chain of progressively developed methods including MLA, LoRA, and SVD, emphasizing how m→26 Jul 2026Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up o…Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up outside the closed labs. Plus, it gives us the freedom to run→7 Aug 2026Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub! This is super cool and motivating. I am really happy to see t…Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub! This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people. Thanks →11 Aug 2026Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days. Their Meta Muse Glimmer m…Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days. Their Meta Muse Glimmer model is a 30B multimodal reasoning model with a Gemma-like a→26 Sept 2026Reasoning from scratch, round number 5! This time, talking about log-probability scoring (also a great fundamental concept for loss function…Reasoning from scratch, round number 5! This time, talking about log-probability scoring (also a great fundamental concept for loss functions like cross-entropy in pre-training and distillation) and s
TechniqueMultimodal5 recent entries15 Jul 2026Interesting surprise drop from Thinky! The Inkling model looks pretty solid on benchmarks, and it has some little surprises in its architect…Interesting surprise drop from Thinky! The Inkling model looks pretty solid on benchmarks, and it has some little surprises in its architecture: - Small conv layers in several places - An RMSNorm for →28 Jul 2026The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks r…The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts. 1. Yes, it looks relatively complicated, but it's essentially a scaled-up prod→11 Aug 2026Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days. Their Meta Muse Glimmer m…Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days. Their Meta Muse Glimmer model is a 30B multimodal reasoning model with a Gemma-like a→26 Aug 2026Now we know: The popular Ox Alpha LLM was GLM-5.3-Flash... Compared to GLM-5.2, this new GLM-5.3-Flash model uses: - a Kimi Linear-style 3:1…Now we know: The popular Ox Alpha LLM was GLM-5.3-Flash... Compared to GLM-5.2, this new GLM-5.3-Flash model uses: - a Kimi Linear-style 3:1 (super*) hybrid attention pattern with 34 Kimi Delta Attent→10 Sept 2026Nice showcase that interesting LLM work can be done on single GPU!Nice showcase that interesting LLM work can be done on single GPU! I extended the GPT-2-style code from @rasbt's 'Build a Large Language Model (from Scratch)' so that it was a 6-expert (2 active) mixt
TechniqueSafety1 recent entries10 Oct 2026Reasoning From Scratch: Reinforcement Learning with Verifiable Rewards (RLVR) round 2. Covering clipped policy ratios, KL loss term, format …Reasoning From Scratch: Reinforcement Learning with Verifiable Rewards (RLVR) round 2. Covering clipped policy ratios, KL loss term, format rewards, and other GRPO tips & tricks. 00:00 Introduction an