Informational Antilocality and the Locality Bias in LLMs

arXiv:2608.27760v1 Announce Type: new Abstract: We consider the ability of transformer-based language models (LLMs) to learn what we call k-antilocal languages, i.e., languages that have no mutual information across any span of k contiguous symbols. We construct such languages with increasing k, finding that LLMs trained on them achieve comparable cross-entropy loss regardless of antilocality, but converge more slowly on more antilocal languages. Our findings support the idea that non-local dependencies are more difficult to learn, but the evidence for this bias comes from learning speed rather than learning success.

Related

• On the 'Induction Bias' in Sequence Models
• Systematic Analysis of Large Language Models and Transformer-Based Machine Translation for English-Tamil and Tamil-English Across Diverse Datasets
• All Languages Matter: Understanding and Mitigating Language Bias in Multilingual RAG

Source: arXiv cs.CL | 2026-08-31