Research
SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learning
arXiv:2608.18319v1 Announce Type: new Abstract: Neural networks trained on nonstationary tasks frequently lose the ability to fit new targets, a phenomenon referred to as loss of plasticity. We identi
arXiv:2608.18319v1 Announce Type: new Abstract: Neural networks trained on nonstationary tasks frequently lose the ability to fit new targets, a phenomenon referred to as loss of plasticity. We identify a novel source of plasticity loss due to the growing anisotropy of weight matrices' singular values during training, and analyze this phenomenon both empirically and theoretically. To mitigate this issue, we introduce SingularClip, a procedure that periodically clips the singular values of all weight matrices. We show that SingularClip performs strongly against baselines across a range of tasks in both continual supervised learning and deep reinforcement learning.
Related
- Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective
- From Order to Distribution: A Spectral Characterization of Forgetting in Continual Learning
- How the Hessian-Spectrum of Neural Networks Depends on Data
Source: arXiv cs.LG | 2026-08-20