Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking

arXiv:2605.08119v1 Announce Type: cross Abstract: Tian (2025) proves a repulsion theorem (Theorem 6) for the matrix B = (widetilde{F}^op widetilde{F} + eta I)^{-1} during the interactive feature-learning stage of grokking: similar features have negative off-diagonal entries B_{jell} , producing an effective repulsive force that drives them apart. However, the theorem does not specify when this mechanism becomes empirically observable, nor whether it leaves a measurable spectral signature in the parameter updates. We test this directly on Tian's modular addition setup ( M = 71 , K = 2048 , MSE loss) and observe a clear structure-mechanism dissociation. The predicted sign rule holds robustly on the top-200 most-similar feature pairs across activations (empirical sign-match rising from 0.865 to 0.985 on sigma = x^2 across 5 seeds, and saturating at 1.000 on sigma = operatorname{ReLU} ). However, the spectral signature in the parameter updates is strongly activation-dependent. With sigma = x^2 , a simple slope detector on the rolling eigengap sigma_2 / sigma_3 of Delta W fires in 15/15 grokking seeds at epoch 174 (IQR [173,174]) and in 0/15 non-grokking controls, with 229 imes late-stage magnitude separation; the spectrum is rank-2. In contrast, with sigma = operatorname{ReLU} , the detector never fires and the spectrum remains effectively rank-1. This dissociation aligns with Tian's Theorem 5 distinction between focused (power-law) and spreading (ReLU) memorization: while the sign structure of B depends only on widetilde{F}^op widetilde{F} , how feature repulsion translates into weight updates critically depends on the activation derivative sigma' .

Source: arXiv cs.AI | 2026-05-12