Infrared Universality of Collective Dynamics across Transformer and State-Space Architectures

arXiv:2608.18592v1 Announce Type: new Abstract: Whether distinct neural architectures develop common collective dynamics remains an open question. Recent analysis of Transformer language models revealed a nearly flat, weakly infrared-enhanced time-scale density of states (TDOS) associated with near-marginal long-memory dynamics. Here we test whether a closely related organization emerges in Mamba, whose selective state-space dynamics provides a fundamentally different microscopic mechanism. Mamba allows relaxation dynamics to be resolved at three levels: the intrinsic spectrum of the learned state-space generator, its input-conditioned selective rescaling, and the collective TDOS of the complete block measured from its Jacobian. These spectra are not identical: selective dynamics and the remaining block transformations substantially reorganize the microscopic relaxation hierarchy. Nevertheless, the full block develops a reproducible slow-mode continuum whose infrared sector becomes progressively better resolved with increasing sequence length. Cumulative analysis yields rho(lambda)simlambda^eta, with the long-sequence Mamba exponent stabilizing near eta_{rm M}simeq-0.17. The corresponding memory dynamics follows K(t)sim t^{-(1+eta)}, close to the marginal 1/t regime. Despite fundamentally different microscopic dynamics, Transformer full-block spectra exhibit closely related infrared organization, with representative exponents of order eta_{rm Tr}sim-0.1. These results separate explicit state-space memory from collective infrared organization and show that distinct sequence architectures can develop closely related near-marginal slow-mode dynamics. They extend infrared collective organization beyond Transformers and provide an independent test of the dynamical structure described by Cognitive Field Theory.

Related

• Universal Time Series Generation with Neural Controlled Differential Equations
• Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime
• Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference

Source: arXiv cs.LG | 2026-08-20