Research
Biologically Inspired Mechanisms for Facilitating Grokking in Multilayer Perceptrons
arXiv:2608.28184v1 Announce Type: new Abstract: Grokking is a delayed transition from memorization to generalization that is often accompanied by substantial reorganization of internal representations
arXiv:2608.28184v1 Announce Type: new Abstract: Grokking is a delayed transition from memorization to generalization that is often accompanied by substantial reorganization of internal representations. This paper studies whether biologically inspired mechanisms, many of which are not commonly incorporated into artificial neural networks, can actively promote this transition by regulating hidden-layer computation at the levels of neuronal activity, response, and effective connectivity. We augment a multilayer perceptron with input gating, structural plasticity, gain modulation, threshold modulation, homeostasis, lateral inhibition, and activation decorrelation, and evaluate these mechanisms through systematic ablations on two established grokking benchmarks: sparse parity and noisy XOR classification. The results show that the mechanisms contribute unequally to generalization. Homeostasis provides the strongest and most consistent benefit, while structural sparsification emerges as the second major mechanism. The remaining biologically inspired mechanisms have smaller or less consistent effects in the present experiments. For both problems, the results support the common principle that explicit regulation of neuron utilization and effective connectivity can improve the emergence of generalizable internal computation. These findings motivate broader investigation of biologically inspired activity regulation and adaptive sparsification, including in large language models, where they may accelerate the development of generalizable representations and reduce the optimization time required for robust generalization.
Related
- The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization
- Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning
- Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokking
- To Grok Grokking: Provable Grokking in Ridge Regression
- Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?
Source: arXiv cs.LG | 2026-08-31