MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs

arXiv:2510.19366v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) scales model capacity through sparse activation, and is becoming an important architecture for large language models (LLMs). However, existing MoE serving systems typically execute all requests under a fixed routing configuration, limiting their ability to exploit heterogeneous computation requirements across requests. Routing top-k, which determines the number of routed experts activated per token, directly controls routed-expert computation and provides a natural mechanism for request-level compute elasticity. Realizing this capability, however, requires finer-grained routing units and efficient runtime execution for heterogeneous routing budgets. We present extsc{MoE-Prism}, a model and system support framework for request-level compute elasticity in MoE serving. extsc{MoE-Prism}decomposes monolithic experts into fine-grained sub-experts to expose denser routing operating points and provides a k-aware serving runtime that effectively serves heterogeneous routing budgets under both throughput-oriented and latency-sensitive workloads. We implement extsc{MoE-Prism} on top of vLLM and evaluate it on three representative MoE models. extsc{MoE-Prism} expands the number of available routing operating points by 4imes, improves offline inference throughput by up to 33.9%, and reduces online serving TTFT under heterogeneous workloads. These results demonstrate practical elastic MoE serving with request-level routing targets.

Source: arXiv cs.CL | 2026-08-11