CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling

arXiv:2610.08312v1 Announce Type: new Abstract: Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters. However, we reveal that such strict geometric constraints trigger an "Orthogonality Dilemma": rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks. In this work, we propose a new replay-free method, called Consolidation and Decoupling LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating Universal Knowledge and Decoupling Task-Specific Knowledge. To achieve this, CoDe-LoRA leverages an adaptive null space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation. Experimental results across four backbones and three CL benchmarks show that CoDe-LoRA achieves the best average accuracy. Our code is available at https://github.com/Estrellajer/CoDe-LoRA.

Related

• Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models
• PALoRA: Projection-Adaptive LoRA for Preserving Reasoning in Large Language Models
• Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
• Decoupling Memory from Context: Structured Memory for Token-Efficient Test-Time Continual Learning
• JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models

Source: arXiv cs.AI | 2026-10-07