Pruning and Distilling Mixture-of-Experts into Dense Language Models
DGX agentarXiv:2605.28207v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory,