Research
A Deeper Analysis of Block-Sparse Featurizers
arXiv:2608.27515v1 Announce Type: cross Abstract: The recently introduced block-sparse featurizer (BSF; Fel et al., 2026) is similar to a sparse autoencoder (SAE), but its atomic unit is a small subsp
arXiv:2608.27515v1 Announce Type: cross Abstract: The recently introduced block-sparse featurizer (BSF; Fel et al., 2026) is similar to a sparse autoencoder (SAE), but its atomic unit is a small subspace (a block of directions) rather than a single direction. It is designed for features that live on low-dimensional manifolds, which are especially frequent in vision. This work studies the BSF's strengths and weaknesses, finding how it still somewhat suffers from classic SAE failure modes, like feature splitting and composition. We propose several architectural changes to the BSF, including a Tournament Top-K selection rule that significantly reduces feature splitting, and we also extend the block paradigm to the crosscoder.
Related
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders
- Fine-Grained Action Recognition with Cross-Attentive Latent Sparse Experts
- Time-varying rPPG signal separation via block-sparse signal model
Source: arXiv cs.CV | 2026-08-31