Research
Beyond Linearity in Attention Projections: The Case for Nonlinear Queries
arXiv:2603.13381v2 Announce Type: replace-cross Abstract: Recent algebraic analysis shows that in decoder-only and encoder-only transformers, the Query projection W_Q may be set to identity without no
arXiv:2603.13381v2 Announce Type: replace-cross Abstract: Recent algebraic analysis shows that in decoder-only and encoder-only transformers, the Query projection W_Q may be set to identity without noticeable performance deterioration. This is possible because attention depends on X only through the products XW_Q, XW_K, XW_V, allowing basis transformations to be absorbed by adjacent layers and propagated through the network. We replace W_Q in R^{d imes d} with a nonlinear residual of the form Q(X) = X + f_heta(X), where f_heta is a bottleneck MLP with d^2 + O(d) parameters. The identity term anchors the nonlinearity to a known-good prior. Experiments on GPT-3 small style models show consistent improvement over the baseline (2.40% lower validation log-loss, 6.81% lower perplexity), comfortably outperforming a model with 12.5% more non-embedding parameters. These results motivate investigation at larger scales and across modalities.
Related
- Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling
- On the Existence of Universal Simulators of Attention
- A Mathematical Explanation of Transformers
- Compressible Softmax-Attended Language under Incompressible Attention
Source: arXiv cs.AI | 2026-04-27