Model Releases
Reasoning From Scratch round 8, the finale! This last lecture in the series covers distillation as an efficient alternative to the previous …
Reasoning From Scratch round 8, the finale! This last lecture in the series covers distillation as an efficient alternative to the previous reinforcement learning training runs. (Planning to take a br
Reasoning From Scratch round 8, the finale! This last lecture in the series covers distillation as an efficient alternative to the previous reinforcement learning training runs. (Planning to take a brief break, but I will be back!) 00:00 Introduction and series recap 01:07 Why distill a reasoning model? 02:51 Soft vs. hard distillation 11:06 On-policy vs. off-policy distillation 18:09 Preparing data for reasoning distillation 22:35 Loading the DeepSeek R1 MATH dataset 28:04 Inspecting reasoning traces and final answers 31:27 Tokenization and chat templates 35:58 Pre-tokenizing the training dataset 39:01 Sequence lengths and the 2,048-token limit 42:38 Creating training and validation splits 45:32 Loading the Qwen3-0.6B base model 46:24 Computing answer-token log probabilities 54:10 Cross-entropy loss for distillation 57:25 Training and validation loss helpers 1:02:03 Implementing the distillation training loop 1:07:26 Running the standalone training script 1:09:41 Plotting training and validation losses 1:11:31 Evaluating checkpoints on MATH-500 1:14:09 Comparing teachers and distillation results 1:16:34 Future directions for reasoning models 1:19:41 Bonus exercises and next steps Media
Related
- Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning
- OISD: On-Policy Internal Self-Distillation of Language Models
- DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models
Source: Sebastian Raschka (X) | 2026-10-17