Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
DGX agentarXiv:2607.21005v1 Announce Type: new Abstract: Most explanations of training instability focus on learning-rate criticality, typically characterized by the Edge of Stability, beyond which optimizatio