The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension

arXiv:2608.28150v1 Announce Type: new Abstract: Which geometry controls the rank complexity of normalized softmax attention? We study maximum-row-ell_1 approximation rank, exactly the least unrestricted rank preserving every bounded vector-valued output. Two sharp worst-case laws isolate support geometry: for fixed d and error arepsilon, spherical self-attention has rank Theta_{d,arepsilon}(min{n,(1+eta)^{(d-1)/2}}), while full-ball geometry adds one radial degree and, for etageeta_0(d,arepsilon) and nge C_d e^{eta/8}, gives Theta_{d,arepsilon}(eta^{d/2}). For a fixed head, row-softmax quotients out row-scalar logit directions: the remaining visible query--key interaction dimension r yields an r/2 per-instance upper law, and bounded constructions show this exponent is minimax sharp. Approximate interaction subspaces incur an explicit residual output error and yield a tolerance-indexed SVD dimension. On an 84-head BERT-base calibration set, we observe modest effective-dimension reductions across many head--temperature settings, together with positive associations with finite constructive rank upper certificates. Together, these results separate support geometry, which sets worst-case temperature scaling, from softmax-visible interaction geometry, which controls per-head approximation complexity.

Related

• Gated MLPs as Symmetry-Broken Rank-1 Bilinear Attention
• Low-Rank Key Value Attention
• On the Invariants of Softmax Attention
• Toward a First-Principles Update Geometry for the Language-Model Head

Source: arXiv cs.LG | 2026-08-31