Local Ai
v0.40.0-rc1: mlx: match publisher tokenizer semantics (#18779)
mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges. Handle empty added tokens and empty Metas
mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges. Handle empty added tokens and empty Metaspace input consistently. Add shared Go/Python reference cases using published tokenizers, pulling missing models directly and failing on errors, plus focused regressions for configuration precedence, byte fallback, and parallel encoding. address comments
Related
- v0.22.1-rc1
- v0.30.0-rc4: ci: windows mlx tuning
- v0.31.1: mlx: tighten up gemma4 moe loading code (#16964)
Source: Ollama Releases | 2026-10-04