Safety
Towards a mathematical theory of superposition
arXiv:2608.27540v1 Announce Type: cross Abstract: We develop a mathematical theory of superposition in neural networks using tools from frame theory and compressed sensing. In our model, a sparse bina
arXiv:2608.27540v1 Announce Type: cross Abstract: We develop a mathematical theory of superposition in neural networks using tools from frame theory and compressed sensing. In our model, a sparse binary vector (x) of active features is encoded through an overcomplete dictionary (W), and feature recovery is performed by applying (operatorname{ReLU}(W^op W x+b)) with an appropriate bias vector (b). We prove several recovery theorems for this model. In the random-support setting, we establish high-probability support recovery for nearly tight, low-coherence dictionaries, with guarantees when the expected sparsity is up to order (d/log n). In the worst-case support setting, we give a sharp and computable criterion for which sparsity levels permit support recovery. We apply this criterion to Gaussian random matrices and equiangular tight frames. For real equiangular tight frames with (n>d+1), we determine the exact recovery threshold in terms of the coherence. The proof of this result for real equiangular tight frames relies on a novel characterization---which should be of independent interest to frame theorists---of the distribution of signs in the Gram matrix.
Related
- A Commutator Framework for Selective Spectral Alignment in Deep Neural Networks
- A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights
- Signal-to-Noise Ratio and Sample Size Govern Representational Alignment in Neural Networks
Source: arXiv cs.LG | 2026-08-31