Safety
SuTRA : Structurally-Unified Tokenization with Root Awareness
arXiv:2608.18087v1 Announce Type: cross Abstract: Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affix
arXiv:2608.18087v1 Announce Type: cross Abstract: Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affixes. This is harmful for morphologically rich Indic languages, where basic units are complex orthographic syllables (aksharas) rather than letters. Frequency-based methods over-fragment words, arbitrarily splitting roots and affixes - a phenomenon we term Morphological Shattering. We propose SuTRA (Structurally-Unified Tokenization with Root Awareness), a morphology-aware algorithm that preserves akshara indivisibility and penalizes merges crossing morphological boundaries. We also release a new morphological segmentation dataset for Hindi, Marathi, and Gujarati. SuTRA reduces shattering, achieving peak gains of +14.7% in morphological alignment (Boundary F1) and +34% in semantic recoverability (Hindi) over BPE. These structural gains yield an average improvement of +8.08 chrF2 in machine translation.
Related
- Probabilistic Language Tries: A Unified Framework for Compression, Decision Policies, and Execution Reuse
- Support sufficiency as action-sufficient compression: a single-cycle rate-regret formulation
- Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
Source: arXiv cs.AI | 2026-08-20