Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

arXiv:2604.25578v1 Announce Type: new Abstract: We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-extsc{Instruct} variants, which surpass the performance of competing models possessing 3--14imes more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights.

Related

• Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models
• HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
• XITE: Cross-lingual Interpolation for Transfer using Embeddings
• Losing our Tail, Again: (Un)Natural Selection & Multilingual LLMs
• Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
• EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

Source: arXiv cs.CL | 2026-04-29