EMO: Pretraining mixture of experts for emergent modularity

EMO is a pretraining approach that uses mixture of experts (MoE) architecture to develop emergent modularity in language models, enabling different experts to specialize in different types of knowledge or tasks. This method, developed by AllenAI, aims to improve model efficiency and interpretability by allowing sparse activation of specialized experts rather than using all parameters for every token. The approach demonstrates how modular structure can naturally emerge during pretraining without explicit architectural constraints.

Source: Hugging Face | 2026-05-08