Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems

arXiv:2610.15968v1 Announce Type: new Abstract: Spiking Transformers achieve energy efficiency by discarding softmax normalization in favor of thresholded, binary attention. However, this removal breaks the thermodynamic framework used to analyze attention networks, as the normalized measure and partition function cease to exist. We restore this theoretical foundation by proving that softmax-free spiking attention operates exactly as the zero-temperature limit of a product of independent two-level systems. This correspondence places spiking attention strictly in the frozen ordered phase of the thermodynamic phase diagram for all inputs. Consequently, the entropic cost of organizing attention collapses, replaced by a Landauer-type physical floor governed by action-potential generation. Furthermore, this ordered-phase pinning mathematically explains the optimization difficulty caused by vanishing surrogate gradients. We validate this theory with forward-relaxation experiments, placing both standard and neuromorphic attention mechanisms on a unified thermodynamic axis.

Related

• Compute-Optimal Scaling Laws for Spiking Language Models
• Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers
• Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling

Source: arXiv cs.LG | 2026-10-14