Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix

arXiv:2610.12744v1 Announce Type: new Abstract: Parameter-efficient continual learning updates a restricted subset of adapter parameters to incorporate sequential tasks while limiting catastrophic forgetting. Gradient attribution is a common rule for selecting that subset, but it requires an additional backward pass over calibration data. We identify a parameter-pool confound in comparisons between initialization-time gradient masks and unrestricted random masks under standard zero-initialized LoRA (B=0): the gradient with respect to A is identically zero (partialL/partial A=0), so gradient masks select strictly from the B matrix whereas unrestricted random masks draw from both A and B. Such a comparison changes both the selection rule and the candidate parameter pool. Isolating the selection criterion by restricting both approaches to the same parameter pool, we find a negative result: a uniform random mask consistently achieves lower loss-based backward transfer than gradient attribution across seven seeds on Qwen2.5-3B. The effect is concentrated on the medical task, which contributes 80% of the aggregate; code and math move in the same direction, and legal shows no reliable difference. This advantage is consistent with a stability--plasticity trade-off under the tested task sequence, in which random masking slightly reduces single-task fitting. On the two discrete-option tasks (Medical and Legal), the mean accuracy-BWT difference is not statistically distinguishable from zero.

Related

• Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
• Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning
• MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning
• Learning When to Adapt

Source: arXiv cs.LG | 2026-10-12