TST works in two phases. In phase 1, which covers the first 20-40% of training, the model reads contiguous bags of k tokens, with input embe…

TST works in two phases. In phase 1, which covers the first 20-40% of training, the model reads contiguous bags of k tokens, with input embeddings averaged within each bag, and predicts the next bag of k tokens using a sum of standard cross-entropy terms. In phase 2, it returns to ordinary next-token prediction for the remainder of the run. The loss curve resolves within a few thousand steps and crosses below the matched-FLOPs baseline for the rest of the run. The entire input-side change is a reshape and a mean over the embedding lookup. The output-side change is a sum of standard cross-entropy terms. No new kernel, no auxiliary head, and no change to the output projection.

Source: Nous Research (X) | 2026-05-13