Entropy-Generated Attention Beyond Softmax and Entmax: Kaniadakis and Reciprocal-Symmetric Abe Operators

arXiv:2602.08216v3 Announce Type: replace Abstract: We derive two attention operators from generalized statistical entropies. Kaniadakis entropy yields an exact full-support normalization whose weights and low-score sensitivities decay algebraically, rather than exponentially as in Softmax or by exact truncation as in entmax. Classical Abe entropy yields an implicit reciprocal-symmetric operator. With q=e^epsilon, the involution qleftrightarrow q^{-1} removes every odd correction about Softmax; we obtain the normalized second- and fourth-order terms, including the deformation of the normalization multiplier. These stationary laws follow from a Fisher-metric Lagrangian on the probability simplex, whose Shannon sector recovers scaled dot-product Softmax. We also give a tangent-gradient test for deciding whether changing the entropy changes the attention profile or only its scale. Renyi and two-parameter Sharma--Mittal entropies retain the Tsallis--entmax inverse-gradient shape, but their global moments make the effective temperature input dependent when the external temperature is fixed. Distinguishing profile-shape equivalence from fixed-parameter operator equivalence separates new normalization shapes from adaptive rescalings and organizes the operators by support, tail behavior, and realization complexity.

Related

• Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
• A Quantum Roadmap for Softmax Attention: Exact Born-Rule Analogs for Softmax Attention on the Probability Simplex
• Forgetful Attention: A Trainable Support-Vector Memory with Certified Selection and Exact Unlearning
• Gradient Boosting within a Single Attention Layer

Source: arXiv cs.LG | 2026-09-04