Enhancing SAE-based Steering via Neighbor Integrated Feature Selection

arXiv:2608.28806v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) disentangle model activations into interpretable features and are widely used for steering large language models. Most existing SAE-based steering methods select features by applying a top- filter based on statistical scores, assuming that higher-scoring features yield stronger steering effects. In this paper, we show that this assumption is often invalid, leading to suboptimal feature selection. Our analysis reveals that effective steering features may be distributed among representationally adjacent, semantically similar groups induced by feature splitting in SAEs. Within such groups, features may exhibit disparate statistical scores despite having comparable steering influence, causing score-based selection to overlook important features. Based on these observations, we propose extsc{Neighbor Integrated Feature Selection} (extsc{NIFS}), a plug-and-play strategy that leverages representation similarity to improve feature selection for steering. We evaluate extsc{NIFS} across multiple SAE-based steering methods and tasks, and demonstrate consistent performance gains over conventional top-k selection.

Related

• C^{2}R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders
• A decodability criterion predicts when hidden-state selection beats majority voting in large language models
• Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders
• Non-linear Interventions on Large Language Models
• Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression

Source: arXiv cs.AI | 2026-09-01