EMO: Pretraining mixture of experts for emergent modularity
EMO is a pretraining approach that uses mixture of experts (MoE) architecture to develop emergent modularity in language models, enabling different experts to specialize in different types of knowledg