Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training

arXiv:2610.20102v1 Announce Type: new Abstract: We investigate two aspects of multilingual pre-training with limited data, namely how language exposure is distributed across English, Dutch, and Chinese, and how that exposure is ordered during training. In particular, we compare balanced multilingual exposure to language-dominant exposure, where one language is exposed with more training data than the others. We also compare mixed training, where multilingual training data is used throughout training, with curriculum training, where monolingual exposure of the dominant language is followed by balanced multilingual exposure. Our results show that multilingual BabyLM performance varies with both the amount and order of language exposure. Mixed training outperforms curriculum learning across all 15 matched configurations, indicating that simultaneous multilingual exposure is more effective than staged exposure in our setting. Several language-dominant mixed training configurations achieve higher scores than the balanced configuration, suggesting that a balanced distribution is not necessarily the highest-performing configuration in our setting. Interestingly, the Dutch-primary configurations achieve the best overall performance.

Related

• Counterfactual Gate-Support Decomposition: Separating Selection from Redistribution in Training-Data Filtering
• Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining
• English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training

Source: arXiv cs.CL | 2026-10-15