Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts

arXiv:2609.01100v1 Announce Type: new Abstract: In current Mixture-of-Experts architectures, routing is performed based on representations dominated by structure shared across all tokens, limiting expert specialization. We show that contrasting each token against an Exponential Moving Average of the layer's hidden states, rather than routing on absolute magnitude, concentrates the routing signal onto a low-dimensional, highly separable subspace. Building on this, we propose the Contrastive Routing Mechanism (CoRM), which scores each expert by the gap between its affinity for the incoming token and its affinity for this shared reference state, interpreted through a distinct per-expert projection. The resulting experts have routing boundaries that align with linguistic structure significantly more than the Top-k baseline. Our experiments show that CoRM improves average zero-shot accuracy by +0.67 to +1.69 points (Top-1) and +1.38 to +1.77 points (Top-2) over standard Top-k MoE baselines on nine zero-shot reasoning benchmarks, at the minimal cost of 2.9% added parameters and 2.6% added FLOPs per token.

Related

• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
• A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models
• PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition
• DEER: Disentangled Mixture of Experts with Instance-Adaptive Routing for Generalizable Machine-Generated Text Detection
• Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models

Source: arXiv cs.CL | 2026-09-02