VORT: Adaptive Power-Law Memory for NLP Transformers

arXiv:2605.08966v1 Announce Type: new Abstract: Standard Transformers impose near-exponential decay on the influence of distant tokens, conflicting with the power-law structure of long-range dependencies in natural language. We introduce the Variable-Order Retention Transformer (VORT{}), a memory architecture in which each ingested token is assigned a learnable fractional order alpha_iin[elta,1] that governs a Grunwald--Letnikov power-law retention kernel. Because the fractional weighted sum is non-Markovian, we approximate it through a sum-of-exponentials (SOE) decomposition computed by Gauss--Laguerre quadrature on a Laplace-type integral representation of the kernel weights. Each exponential component admits a one-step Markovian recurrence at O(Sd_v) per step, where S=O(log(T/arepsilon)) terms suffice for arepsilon-uniform accuracy on horizon [1,T]. Retrieval is keyed and associative via a linear-attention accumulator with an exact O(KSd_phi d_v) -per-step recurrence. Four results are established: (i) an SOE approximation theorem with geometric convergence rate from the analyticity of the integrand after a log-change of variables; (ii) a quantisation bound valid on [elta,1] with correct analysis near alpha=0; (iii) a direct L^2 energy argument (Proposition) showing that for alpha>1/2 any mixture with fixed minimum decay rate Lambda>0 incurs L^2([1,T]) error at least N_alpha(T)-C(Lambda)oinfty, with the Lambda-dependence made explicit; and (iv) linear convergence of a gradient plasticity rule under the Polyak--L{}ojasiewicz condition. Two synthetic experiments confirm the architectural advantage: a Zipf-distributed retrieval benchmark and an entity label-copy task with uniform lag distribution, the latter ruling out prior-matching as an explanation for the power-law kernel's advantage.

Source: arXiv cs.LG | 2026-05-12