One Spectrum, Two Resources: Data-Memory Scaling in Autoregressive Prediction

arXiv:2609.13500v1 Announce Type: cross Abstract: How much learned memory is needed to benefit from more data? We show that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source. Each coordinate contributes its query probability times the squared radius of its unknown logit. Writing mu for the resulting energy spectrum, we prove the minimax law mathfrak R^*_{rm value}(n,B)asymp_R Phi_mu(n^{-1})+Phi_mu(au_B), Phi_mu(t)=intmin{x,t},mu(mathrm dx), for n prediction blocks and a learned state with at most 2^B values. Data set the resolution 1/n; memory sets the level au_B reached by optimal bit allocation. The complete curve also recovers the positive spectrum. Energy-dimension pairing is essential: two causal sources with identical block-energy and block-dimension marginals have different data and memory exponents. A masked query-key attention head learns the route and values, realizing the law with explicit routing, format, and arithmetic errors. Further results give exponent-adaptive allocation, finite-precision realization, and compute-precision laws under two-sided arithmetic assumptions. Experiments recover the data-memory collapse and coupling exponents, explain the routing and allocation mechanisms, and examine weight-only quantization across six pretrained-model scales.

Related

• How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions
• Measuring Memory and Generalization as Separable Geometric Channels: The Topo^2 Framework
• How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
• Data Provenance for Image Auto-Regressive Generation

Source: arXiv cs.AI | 2026-09-15