A Faster Path to Continual Learning

arXiv:2604.11064v1 Announce Type: cross Abstract: Continual Learning (CL) aims to train neural networks on a dynamic stream of tasks without forgetting previously learned knowledge. Among optimization-based approaches, C-Flat has emerged as a promising solution due to its plug-and-play nature and its ability to encourage uniformly low-loss regions for both new and old tasks. However, C-Flat requires three additional gradient computations per iteration, imposing substantial overhead on the optimization process. In this work, we propose C-Flat Turbo, a faster yet stronger optimizer that significantly reduces the training cost. We show that the gradients associated with first-order flatness contain direction-invariant components relative to the proxy-model gradients, enabling us to skip redundant gradient computations in the perturbed ascent steps. Moreover, we observe that these flatness-promoting gradients progressively stabilize across tasks, which motivates a linear scheduling strategy with an adaptive trigger to allocate larger turbo steps for later tasks. Experiments show that C-Flat Turbo is 1.0imes to 1.25imes faster than C-Flat across a wide range of CL methods, while achieving comparable or even improved accuracy.

Related

• Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations
• IMU: Influence-guided Machine Unlearning
• Sharpness-Aware Surrogate Training for On-Sensor Spiking Neural Networks
• Catalyst: Out-of-Distribution Detection via Elastic Scaling

Source: arXiv cs.CV | 2026-04-14