BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
DGX agentarXiv:2605.29379v1 Announce Type: new Abstract: We present BrahmicTokenizer-131K, a 131,072-vocabulary byte-level BPE tokenizer that closes the Brahmic compression gap at the 131K-vocabulary class whi