Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

arXiv:2608.24814v1 Announce Type: new Abstract: We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.

Related

• Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
• AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
• A Self-Attentive Meta-Optimizer with Group-Adaptive Learning Rates and Weight Decay

Source: arXiv cs.LG | 2026-08-26