Formalizing Forgetting: Theory of Forgetting for Online Continual Learning

arXiv:2610.14787v1 Announce Type: new Abstract: Online continual learning (OCL) requires a learner to acquire a sequence of tasks in a single pass over the data, and catastrophic forgetting of earlier tasks remains its central challenge. Despite the many methods proposed to mitigate forgetting, the theoretical understanding of forgetting in OCL is still limited: existing analyses mostly focus on linear, overparameterized, or neural tangent kernel regimes, and it remains unclear which factors govern forgetting and how they affect it. To fill this gap, we analyze single-pass Shuffle SGD under strongly convex and general convex objectives and derive explicit upper bounds on the expected final forgetting. Further analysis of these bounds shows how the per-task data amount, task similarity, and task ordering enter them, and that their effects can differ across objective classes. Moreover, experiments with neural networks on Split-CIFAR-100 and PMNIST show that the trends suggested by our theory also appear in practice, indicating that these insights extend beyond the convex setting.

Related

• On the Theory of Continual Learning with Gradient Descent for Neural Networks
• Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weights
• The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning
• From Order to Distribution: A Spectral Characterization of Forgetting in Continual Learning
• Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective

Source: arXiv cs.LG | 2026-10-14