Canalization Before Generalization: Grokking as a Dynamical Probe

arXiv:2608.25813v1 Announce Type: new Abstract: For overparameterized neural networks, many solutions can fit the training data equally well while behaving very differently on unseen samples. Grokking separates training fit from visible generalization, providing a window for studying how this selection develops during training. We sweep short, fixed-duration weight-decay (WD) pulses across this plateau and measure how they shift later generalization time. Across three grokking tasks, these shifts are unordered early in the plateau but later form a stable dose ordering, with stronger WD increases leading to earlier generalization and stronger WD decreases leading to later generalization. This ordering emerges before visible generalization in all three tasks. Meanwhile, test-loss barriers between perturbed and baseline generalization checkpoints collapse toward zero while the ordered timing effects persist. We call this combination of increasingly constrained solution selection and persistent dose-ordered timing sensitivity the canalization of function selection.

Related

• A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
• Thermodynamic Weight Decay: Exploring Grokking Acceleration via Attention Specific Heat
• Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory
• The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization

Source: arXiv cs.LG | 2026-08-27