Research
Effective Does Not Mean Useful: Conditional Functional Substitutability for Redundancy and Scaling in Transformers
arXiv:2609.39259v1 Announce Type: new Abstract: Modern neural networks scale predictably, yet the mechanisms behind these regularities remain unclear. Neural redundancy is typically characterized by c
arXiv:2609.39259v1 Announce Type: new Abstract: Modern neural networks scale predictably, yet the mechanisms behind these regularities remain unclear. Neural redundancy is typically characterized by component importance or representational similarity, both indirect proxies. We view redundancy as an input-conditioned, dynamic relation: intermediate computational states are functionally redundant when they induce similar downstream responses. We introduce Conditional Functional Substitutability (CFS) to directly characterize such functional substitution. CFS exposes functional relations and reduction potential missed by conventional importance- and similarity-based measures. Across modalities and Transformer families, CFS reveals systematic functional reorganization with scale. Controlled scaling further shows that performance gains need not track growth in substitutability, while fixed-capacity models with more independent functional structure perform better, providing a functional account of diminishing returns. Predicted CFS further enables dynamic computation with a better performance--computation trade-off than importance-based component selection, suggesting new directions for redundancy-aware computation and more efficient model scaling.
Related
- Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
- Individual Parameters in Weight-Sparse Transformers Appear Interpretable
- EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers
Source: arXiv cs.AI | 2026-10-01