Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation

arXiv:2610.20247v1 Announce Type: new Abstract: Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additional per-token computation. Given this extra capacity, does adding experts reduce what each additional language costs? To answer this question, we propose a repetition-aware joint scaling law relating number of active parameters, training tokens per language, number of experts, and language coverage. Fitted on 26,300 per-language loss observations from 64 pretrained configurations, our law reduces prediction error on held-out configurations by 28% relative to the strongest evaluated baseline. At matched number of active parameters and training tokens per language, 32 experts reduce the loss penalty of expanding from 12 to 200 languages by 83% compared with a dense model. Our law further shows that broader language coverage favors more experts across a wider range of compute budgets, guiding capacity allocation as multilingual models scale.

Related

• Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
• GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
• Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
• A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models

Source: arXiv cs.CL | 2026-10-15