Dynamic-Programming-Guided Hierarchical BPE and Empirical Analysis of Vocabulary Pruning
arXiv:2609.06898v1 Announce Type: cross Abstract: Byte Pair Encoding (BPE) constructs vocabularies through greedy pair merging, but the resulting merge order does not necessarily allocate a fixed mode