Subword boundaries are the second meaningful effect. Adding end-of-subword markers as input embeddings produces a large gain throughout trai…

Subword boundaries are the second meaningful effect. Adding end-of-subword markers as input embeddings produces a large gain throughout training (H3): end-boundaries leak future bytes (whitespace always follows an end-boundary, for example) and simplify the next-byte prediction task. Start-of-subword boundaries cannot leak the future, and they also help. When start-boundaries are provided only during the first 50k training steps and removed thereafter for both training and validation, the improvement persists; end-boundaries do not survive the same intervention. One reading is that start-boundaries supply a morphological inductive bias (H4), while end-boundaries supply a near-term prior the model becomes dependent on.

Source: Nous Research (X) | 2026-05-21