Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility

arXiv:2610.20139v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) have emerged as a promising approach to mechanistic interpretability by decomposing language-model activations into sparse, potentially interpretable features. However, differences in design assumptions, evaluation objectives, and experimental settings complicate conclusions about their effectiveness. We present a critical survey of 138 studies on SAEs in natural language processing. Our contributions are fourfold: (1) we propose a task-grounded taxonomy connecting dictionary design, feature operations, and NLP functions; (2) we examine how sparsity allocation, geometry, data coverage, and inference affect feature recoverability; (3) we assess evidence for semantic interpretability, mechanistic faithfulness, and behavioral utility; and (4) we synthesize findings across multilingual processing, retrieval, reasoning, classification, and behavioral regulation, highlighting the roles of objectives, baselines, supervision, and intervention pipelines. Through comparative analyses of published results, we reconcile apparent disagreements, identify unresolved questions, and propose directions for more rigorous evaluation. An anonymous research register https://anonymous.4open.science/r/sae-research-register-C331/register.html accompanies the survey.

Related

• Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders
• SharedSAE: One Feature Dictionary Across Language Models
• Parts-of-Speech as Emergent Categories in SAE Latent Space
• SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization
• C^{2}R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders

Source: arXiv cs.CL | 2026-10-15