BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
DGX agentarXiv:2607.23319v1 Announce Type: new Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and pro