A Function-Space Approach to the Statistical Mechanics of Learning Dynamics

arXiv:2609.09589v1 Announce Type: new Abstract: Deep neural networks exhibit regular macroscopic behavior despite highly nonlinear dynamics in vast parameter spaces. We develop a statistical-mechanical description of learning directly in function space, treating parameter configurations as microscopic realizations and functions with their dynamical operators as macroscopic variables. For mean-squared loss, the exact error dynamics are governed by the learning operator (M=JJ^ast). Combining the dynamical Boltzmann weight of the conditional stochastic dynamics with the parameter-space density of states, whose local curvature defines a statistical operator (B), and integrating over local fluctuations yields $ Phi_{fluc}(M;B)=frac{sigma_xi^2}{2}loget(M^{-1}+B)+const. $ At fixed spectrum, this term is rotationally stationary when ([M,B]=0), is minimized by pairing large eigenvalues of (M) with small eigenvalues of (B), and generates a local restoring contribution against rotational mismatch. For ReLU-type function spaces under mild stable statistical conditions, (B=sigma_xi^2L^astmathcal K L), where (L) measures coarse-grained second-order structure. Thus the low-(B) sector corresponds, up to bounded anisotropy of (mathcal K), to low structural curvature, implying a preference for faster relaxation along smooth, data-adaptive directions. These results identify function space as a natural macroscopic level for studying stable collective organization in learning.

Related

• Human-Centered Learning Mechanics: A Dynamical Framework for Entropy-Regulated Representation Learning
• Wave Function Backpropagation with Explicit Temporal-Interval Dynamics
• Muon Dynamics as a Spectral Wasserstein Flow

Source: arXiv cs.AI | 2026-09-11