The Price of Token Boundaries: Compression Certificates and Prediction
DGX agentarXiv:2609.35869v1 Announce Type: new Abstract: Pre-tokenisation restricts which text fragments can become prediction units, but its compression cost is obscured when tokenisers are compared only unde