Towards Open-Ended Visual Scientific Discovery with Sparse Autoencoders

arXiv:2511.17735v2 Announce Type: replace Abstract: Foundation models in several scientific domains, including visual domains, learn representations that capture complex semantics from their respective fields. Despite this, most existing applications focus on pre-specified concepts or targets, excelling at confirmation but not open-ended discovery of unknown patterns. We study whether sparse autoencoders (SAEs) with heuristic feature ranking can provide a practical methodology for surfacing candidate visual features from vision foundation models for scientific discovery without concept-specific supervision. We evaluate this approach in three stages: (1) general-purpose concept rediscovery, (2) domain-specific concept rediscovery, and (3) question-driven feature ranking. On ADE20K, SAEs outperform baseline methods (k-means, PCA, SemiNMF) across most downstream metrics for concept rediscovery. On FishVista, the same approach recovers fine-grained fish anatomical features without concept supervision. Finally, on Heliconius butterflies, a heuristic based on the Mann-Whitney U statistic consistently surfaces features corresponding to real diagnostic keys between mimic subspecies. Together, these results show that SAEs with heuristic ranking can efficiently learn and surface semantic structure in vision foundation model representations. We view this as a methodological prerequisite for open-ended visual scientific discovery: the present experiments validate feature surfacing and ranking through rediscovery, while prospective discovery of previously unknown scientific phenomena remains future work.

Related

• GeoSAE: Geometric Prior-Guided Layer-Wise Sparse Autoencoder Annotation of Brain MRI Foundation Models
• Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models
• Sparse Autoencoders are Topic Models

Source: arXiv cs.CV | 2026-10-01