Model Releases
Sparse Token Routing in Efficient Transformers
arXiv:2608.20632v1 Announce Type: new Abstract: Efficient-transformer research often motivates token pruning and adaptive computation with the claim that not all tokens require equal computational eff
arXiv:2608.20632v1 Announce Type: new Abstract: Efficient-transformer research often motivates token pruning and adaptive computation with the claim that not all tokens require equal computational effort. We test this claim end to end using SEWN, a two-stream Transformer that routes tokens through either lightweight or full-capacity processing using a learned gate. Across our experiments, routing introduces negligible accuracy change relative to parameter-matched baselines, while the gate's token-importance signal depends critically on how it is learned. A static lexicon-seeded prior fails a counterfactual faithfulness test on BoolQ, whereas a fully contextual gate achieves highly significant separation (p<10^{-10}) on both evaluated tasks without changing task accuracy.
Related
- STS: Efficient Sparse Attention with Speculative Token Sparsity
- Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving
- Less Tokens, Better Forecasts: Sparse Residual Routing for Efficient Weather Prediction
- The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
Source: arXiv cs.CL | 2026-08-24