Federation of Experts: Communication Efficient Distributed Inference for Large Language Models

arXiv:2605.06206v2 Announce Type: replace Abstract: Mixture of experts has emerged as the primary mechanism for making Large Language Models (LLMs) computationally efficient. However, in distributed settings, communicating token embeddings between experts is a significant bottleneck. We present the novel Federation of Experts (FoE) architecture. FoE restructures the MoE block of a transformer layer into multiple MoE clusters. Each cluster is responsible for only one of the KV heads and expert parallelism is applied between those experts. Between clusters, a sum synchronizes the post-attention residuals, which then drives routing and dispatch for the next MoE block. In a single-node setting, FoE completely eliminates all-to-all communication as all experts within a group are contained on the same GPU. In multi-node settings, FoE confines all-to-all communication to the intra-node fabric, thus significantly reducing communication overhead. An implementation of FoE finds that on LongBench, FoE significantly improves inference throughput and latency in both single-node and multi-node settings, reducing end-to-end prefill latency by up to 5.88x, TTFT by 3.66x, and TPOT by 1.53x. It does so while achieving comparable generation quality to a mixture of experts model of the same size and training configuration.

Related

• CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving
• Fast MoE Inference via Predictive Prefetching and Expert Replication
• LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers
• Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference
• BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

Source: arXiv cs.LG | 2026-09-30