Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits

arXiv:2512.14338v4 Announce Type: replace Abstract: Many learning problems are organized by group symmetries. While invariance is often imposed through architectures or group averaging, we ask when it can emerge from training on a finite random subset of an orbit. We study this question in classical Hopfield networks, where strict memorization can be expressed as a linear margin problem. Reparameterizing minimization of energy flow (MEF) as an exponential loss connects gradient descent to the corresponding minimum-norm hard-margin memorizer. Our main result shows that, for independent uniform samples from any finite permutation orbit, the exact sample hard-margin support vector machine (HSVM) concentrates exponentially around the invariant full-orbit HSVM. Consequently, an orbit-size-independent polynomial number of samples suffices both for approximate parameter invariance and for simultaneous memorization of every orbit element; directional convergence transfers this conclusion asymptotically to MEF gradient descent. For graph-isomorphism orbits, we characterize the invariant parameters as a three-dimensional subspace and show that every such orbit is memorizable. For cliques of fixed linear density, additional symmetry sharpens the uniform memorization bound to O(v^4log(1/elta)), exponentially smaller than the orbit size. Together with experiments across several learning rules, these results give a finite-sample account of how optimization bias can recover symmetry from partial group-structured data.

Related

• Beyond ReLU: Bifurcation, Oversmoothing, and Topological Priors
• SILVA Networks as Structured Implicit Layers and Vector Attractors via Dynamic Interaction Fields
• Learning Latent Graph Geometry via Fixed-Point Schrodinger-Type Activation: A Theoretical Study

Source: arXiv cs.LG | 2026-08-17