Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
DGX agentarXiv:2508.04796v3 Announce Type: replace-cross Abstract: Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines. Standard algorithms for learning tokenizers rely on fr