Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
DGX agentarXiv:2606.14122v2 Announce Type: replace Abstract: Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare