High-probability guarantees for linear accessibility in feature superposition

arXiv:2609.09556v1 Announce Type: cross Abstract: Neural networks can leverage feature superposition to encode more concepts than dimensions, but cross-feature interference constrains the linear accessibility of simultaneously active features. By framing linear accessibility as a compressed sensing problem, we derive high-probability bounds for fixed supports under subgaussian noise, proving the sufficient dimension scales linearly (d=O_{arepsilon}(k log m)) rather than prior worst-case quadratic limits. We then validate these bounds across system parameters through Gaussian-tail approximations. These results quantify the geometric constraints of the linear representation hypothesis, providing a framework for evaluating sparse autoencoders, compositional generalization, and neural interpretability.

Related

• Compressed Computation under L^4 Loss is likely Computation in Superposition
• Feature Superposition in Neural Networks: From Theory to Practice
• Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models

Source: arXiv cs.LG | 2026-09-10