Training-free hack to reduce MoE reasoning tokens by 8.5% without losing accuracy but increasing Active parameters per token in MOE (Qwen 35B A4B+)

Hey everyone, Wanted to share a short paper we just published exploring a simple but surprisingly effective optimization for sparse MoE reasoning models. The problem: MoE reasoning models (especially the newer Qwen generations) tend to "overthink" — they burn through tons of reasoning tokens to arrive at answers they could have reached much sooner. This drives up inference costs and latency without improving accuracy. The idea: Instead of retraining anything, we just tweak the router at runtime. Specifically, we expand the expert selection budget (N≥KN≥K) only in the late transformer layers, with a linear decay factor applied to the extra experts. Early layers stay untouched. So Qwen 3.6 35B A3B becomes Qwen 3.6 35B A4B+ ! What we found — "Succinct Convergence": When you give the model more expert capacity at the decision-critical final layers, it stops rambling. It reaches the same correct answer via significantly shorter reasoning trajectories. Results on full MMLU-Pro (714 questions, Qwen3.6-35B-A3B): 📉 8.5% reduction in mean reasoning tokens ⚡ 10.9% drop in latency (p=6.5×10−6) 🎯 Accuracy unchanged (84.5% vs 84.0% native, p=0.77 — statistically indistinguishable) 🆓 Zero training cost — pure inference-time routing modification Ties-only paired comparison on 577 common correct answers. Why this matters for local/self-hosted setups: If you're running MoE models at scale, this is essentially free throughput. Same hardware, same model weights, more queries served per hour. The "layer-scoped" part is key — you're not just burning FLOPs everywhere, you're concentrating compute where the model actually makes its final decisions. Links: 📄 Paper: https://zenodo.org/records/22255483 Questions for the community: Has anyone else noticed this "overthinking" behavior scaling with MoE size? Curious if the same trick transfers to larger Qwen MoE variants or DeepSeek-V4 style architectures. For those running vLLM/SGLang — any thoughts on implementing layer-specific KK values without breaking batch efficiency? That's the main engineering hurdle I see for production deployment. Would love feedback on whether the linear decay factor for expert expansion is the right approach, or if something like softmax-gated decay would be cleaner. submitted by /u/Specific-Tax-6700 [link] [comments]

Source: r/LocalLLaMA | 2026-09-03

Related