RoPE attention is an exact forward-pass gradient step with softmax intact

arXiv:2609.06685v1 Announce Type: cross Abstract: We derive an exact gradient-step representation of the RoPE-softmax forward pass. For every deterministic RoPE-softmax attention head with arbitrary affine projection weights, we construct a query-dependent effective matrix Delta M_i satisfying y_i = mu_i + u_i^op Delta M_i, where mu_i is the uniform mean of the attended values and u_i is the augmented query input. The construction applies the classical exponential divided difference rho = phi_1 to retain the softmax exactly. Its positive coefficients give a unit gradient-step representation on a query-conditioned quadratic objective. The same function connects the RoPE generator to exact positional finite differences. We derive a tokenwise formula for the error of reusing one query's matrix and prove that a nonconstant finite-cache head cannot admit a globally exact affine query readout. Reconstruction checks and frozen-reuse calibration on one pretrained Qwen2.5-0.5B layer verify the representation and quantify the correction required when one query's matrix is reused.

Source: arXiv cs.LG | 2026-09-09