Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders

arXiv:2505.22255v4 Announce Type: replace-cross Abstract: Sparse Autoencoders (SAEs) decompose language-model activations into sparse, interpretable features, but standard encoders usually treat the latent dictionary as a flat set of independent coordinates, leaving hierarchy and feature interactions to emerge only implicitly. We propose KronSAE, a design that factorizes the latent space into heads and forms post-latent features as pairwise compositions of lower-dimensional pre-latents using mAND, a differentiable AND-like interaction. This imposes a compositional co-activation prior while remaining compatible with standard SAE objectives and variants such as TopK, Matryoshka, and Switch SAEs. KronSAE matches strong baselines on the EV-FLOPs frontier, improves interpretability of the latents, better captures the underlying correlated feature structure, and reduces encoder computational cost as an additional benefit. Code is available at https://github.com/corl-team/kronsae.

Related

• Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder Features
• Enhancing SAE-based Steering via Neighbor Integrated Feature Selection
• SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization
• C^{2}R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders
• Feature Rivalry in Sparse Autoencoder Representations: A Mechanistic Study of Uncertainty-Driven Feature Competition in LLMs

Source: arXiv cs.CL | 2026-09-01