We ran a large-scale distillation attack on the Kimi K3 technical report by reading it in parallel at the Hugging Face Journal Club :) https…

We ran a large-scale distillation attack on the Kimi K3 technical report by reading it in parallel at the Hugging Face Journal Club :) https://youtu.be/MW8-kqd2SD8?si=jSKDogcUWbJ8N2k7 Our main takeaway is that there doesn't appear to be any "secret sauce" behind frontier performance. Instead, K3 is a demonstration of how many difficult algorithmic and infrastructure decisions have to work together (distillation, RL scheduling, environments, quantisation, MoE sharding etc) to push a model to the frontier. Here’s what we learned: • Frontier post-training increasingly looks like expert training followed by distillation. K3 trains specialists across three domains and three reasoning-effort levels, then distils the resulting nine experts back into a single checkpoint using multi-teacher OPD • Reasoning effort is treated as a trainable capability. Token budgets are estimated from the SFT model, and a stage-wise curriculum anneals from long to short rollouts to produce low-, high- and max-effort experts. • Partial rollouts keep expensive RL infrastructure busy. Completed trajectories can trigger immediate updates, while unfinished ones are carried into later iterations and reprioritised through a priority queue. Per-token corrections help control the resulting off-policy behaviour. • The reward model is itself an agent. Rather than relying only on fixed or binary rewards, it generates task-specific rubrics and scores responses against them on the fly. • The environment interface is designed for composition. A unified, white-box abstraction allows the same training system to operate across different agent harnesses like Codex, CC, Hermes instead of specialising around a single one. • The systems work is as interesting as the algorithmic work. GPUs dynamically switch between training and inference, rollout concurrency responds to KV-cache pressure, and reference models can live on CPU until needed. • Quantisation is part of training—not merely deployment. K3 uses QAT for weights and activations, while matching the trainer and inference server’s quantisation schemes to reduce train–inference mismatch. • The chat template is novel and introduces an extensible token markup language is intended to accommodate new tools and modalities without repeatedly redesigning the template. • Synthetic task generation is becoming increasingly autonomous. Agents traverse a knowledge graph and search the web to generate training tasks at scale, although the report leaves some details of their integration underspecified. (This part was very cool)

Source: Clem Delangue (X) | 2026-07-29