TST has two moving parts that can be ablated independently: the input-side averaging and the output-side multi-hot cross-entropy. Both, on t…

TST has two moving parts that can be ablated independently: the input-side averaging and the output-side multi-hot cross-entropy. Both, on their own, outperform the matched-FLOPs baseline. Their effects combine approximately additively when used together. We read this as evidence that TST is not a single trick with two knobs but two distinct mechanisms that happen to be compatible. The output-side mechanism is structurally a form of multi-token prediction with weight sharing across positions. The input-side mechanism is harder to place. One reading is that averaging contiguous embeddings acts as a regularizer on the embedding table. Another is that phase 1 is a cheap form of pre-pretraining on a coarser version of the same corpus, a stage of training conducted on a simpler distribution that leaves the model better prepared to learn from natural language in phase 2.

Source: Nous Research (X) | 2026-05-13