Stiefel Attention: When the Geometry of Transformer Projection Matrices Dominates Optimizer Choice---and When It Does Not
arXiv:2609.19363v1 Announce Type: new Abstract: The query and key projections WQ,WK in attention are almost always trained by Euclidean optimizers with no constraint on their geometry. We constrain th