From Tokens to Concepts: Leveraging SAE for SPLADE

arXiv:2604.21511v1 Announce Type: cross Abstract: Learned Sparse IR models, such as SPLADE, offer an excellent efficiency-effectiveness tradeoff. However, they rely on the underlying backbone vocabulary, which might hinder performance (polysemicity and synonymy) and pose a challenge for multi-lingual and multi-modal usages. To solve this limitation, we propose to replace the backbone vocabulary with a latent space of semantic concepts learned using Sparse Auto-Encoders (SAE). Throughout this paper, we study the compatibility of these 2 concepts, explore training approaches, and analyze the differences between our SAE-SPLADE model and traditional SPLADE models. Our experiments demonstrate that SAE-SPLADE achieves retrieval performance comparable to SPLADE on both in-domain and out-of-domain tasks while offering improved efficiency.

Related

• Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models
• Sub-Token Routing in LoRA for Adaptation and Query-Aware KV Compression
• On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stability
• Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMs

Source: arXiv cs.CL | 2026-04-24