Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning

arXiv:2610.21682v1 Announce Type: cross Abstract: Reinforcement learning (RL) can become off-policy due to training-inference mismatch, mini-batch updates, and asynchronous training.The resulting mismatch is especially harmful in Mixture-of-Experts (MoE) models, where rollout--training discrepancies can change routing decisions and lead to router drift. Rollout Routing Replay (R3) mitigates this mismatch by reusing expert selections from the rollout stage during training. Our analysis shows that routing replay introduces gradient bias by losing contributions from experts selected by the training router but not the rollout router. To address this, we propose Router Drift Control (RDC), which uses routing disagreement between rollout and training as a signal of off-policy staleness and selectively rejects high-drift responses. The sequence-level filtering avoids gradient bias introduced by replay while preserving native expert selection and the associated learning signals for retained responses. Experiments show that RDC achieves larger gains as policy staleness increases and remains effective in asynchronous RL. In asynchronous training, RDC combined with Truncated Importance Sampling (TIS) improves the average peak validation accuracy across AIME 2024 and AIME 2025 by 5.5 points over TIS and 4.5 points over R3.

Related

• VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training
• Diagnosing Training Inference Mismatch in LLM Reinforcement Learning
• R^2PO: Decoupling Rollout and Inference Policies for LLM Reasoning
• ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
• Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization

Source: arXiv cs.AI | 2026-10-16