Model Releases
I just built a mini Kimi-K3 from Scratch under 250$. Already beats GPT-2 (124M)!
I pre-trained a 1.02-billion-parameter on Kimi K3 replica trained on 5.00 billion decontaminated tokens for $250. This model has 1.02 billion parameters, of which 145 million are active per token. It
I pre-trained a 1.02-billion-parameter on Kimi K3 replica trained on 5.00 billion decontaminated tokens for $250. This model has 1.02 billion parameters, of which 145 million are active per token. It is roughly one two-thousandth of K3 by total size. It saw 5,000,003,584 tokens, which is a rounding error against the corpora frontier models are trained on. It has never been instruction-tuned, and it has only ever done one thing: predict the next token. What it does have is K3's architecture: - Kimi Delta Attention, Gated MLA, Attention Residuals - LatentMoE with the same aux-loss-free balancer - Same activation function with the same two constants - K3's own 163,840-token tokenizer, unmodified. I report a 33.4% HellaSwag which beats the GPT-2 124M score of 28% Read the entire tutorial here: https://books.vizuara.ai/book/pretraining-a-mini-k3 submitted by /u/OtherRaisin3426 [link] [comments]
Related
- Built and released BetterGPT-150M – A compact 150M parameter completion model (+ live HF Space demo)
- I trained a 1B-parameter LLM from scratch on 20B tokens for about $200
- Kimi K3: Open Frontier Intelligence
Source: r/LocalLLaMA | 2026-08-20