Calibrating Lightweight Sparse Autoencoder Feature Steering

arXiv:2506.12576v3 Announce Type: replace Abstract: Sparse autoencoders (SAEs) can enable inference-time topic steering by modifying latent feature activations, but existing steering methods often fail when target-aligned features are not identified or are modified at the wrong scale. We introduce extsc{ContrastiveSteer} to address these respective failure modes. First, features are scored by how much more strongly they activate on target-domain text than on general text. Second, steering strength is set using a model-specific calibration. We also introduce contamination, a complementary heuristic that measures whether post-steering activations emphasize weakly aligned features, thus indicating off-target or nonsensical outputs. In our evaluations, across multiple LLM families and SAEs, extsc{ContrastiveSteer} improves topic alignment across domains up to 3.9imes over unsteered and steering baselines, and raises target-domain classification by up to 93%. Our activation-scale normalization method also recovers at least 93% of optimal domain-score gains across all tested model families, and contamination distinguishes successful steering, under-steering, and over-steering collapse.

Source: arXiv cs.CL | 2026-09-22