Model Releases
No Equivariant Architecture Covers All Equivariant Attention
arXiv:2608.30417v1 Announce Type: new Abstract: We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group G, then G can o
arXiv:2608.30417v1 Announce Type: new Abstract: We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group G, then G can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many Zariski-irreducible components in a reduced parameter space, and any single architecture covers at most one. For G=D_4 acting on C copies of the regular representation as the token feature space, we show that there are Omega(C^{64}) components for eight attention heads.
Related
- Group-Algebraic Tensors: Provably-optimal Equivariant Learning and Physical Symmetry Discovery
- The Symmetries of Three-Layer ReLU Networks
- Geometric Analysis of Token Selection in Multi-Head Attention
Source: arXiv cs.LG | 2026-09-01