v0.40.0-rc1: mlx: match publisher tokenizer semantics (#18779)
mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges. Handle empty added tokens and empty Metas