Today we release a study on decoupling the benefits of subword tokenization for language model training, by simulating each suspected benefi…
Today we release a study on decoupling the benefits of subword tokenization for language model training, by simulating each suspected benefit one at a time inside a 1.7B byte-level pretraining pipelin