Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute

arXiv:2610.02584v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) models combine learned routing with selection from a large expert pool. We isolate the contribution of dynamic expert combination using a dense analogue: K SwiGLU experts, all active for every token and combined by a softmax gate, at fixed total FFN width. With no larger pool or discrete selection, the dense baseline is the K=1 case. Validation loss varies non-monotonically with K: K=2 improves over the baseline by 0.0048, whereas K=4 and K=6 worsen it by 0.0053 and 0.0197, respectively. Routing generally remains soft and expert usage balanced, except in the first layer of the K=4 model, where routing is nearly one-hot. Forcing this gate to uniform increases loss by 2.4 nats on a diagnostic subset, indicating that its concentrated routing is functionally important. We further show that the architecture is exactly a dense SwiGLU with token-dependent, simplex-constrained group scaling and contains the dense baseline in its function class. These single-run results suggest a granularity sweet spot for always-active, softly gated FFNs at fixed width, with K=2 performing best among the configurations tested.

Related

• MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
• Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
• Boundary Mass and the Soft-to-Hard Limit in Mixture-of-Experts
• You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

Source: arXiv cs.LG | 2026-10-05