Functional Subnetwork Localization for Efficient Continual Learning in Large Language Models

arXiv:2610.21537v1 Announce Type: new Abstract: Biological intelligence continually adapts to new information while retaining prior skills through localized plasticity. In contrast, continual learning in large language models (LLMs) remains vulnerable to catastrophic forgetting. A promising direction is to identify functionally specialized subnetworks within LLMs that enable targeted and resource-efficient adaptation. Using gradient-attribution profiling, we localize cognition-associated functional subnetworks exhibiting extremely low mutual overlap. Building on this finding, we propose Function-isolated Memory-Efficient Optimization (CiMEO), a continual learning framework that routes each training example to its corresponding functional subnetwork, restricting parameter updates and optimizer states to selected regions. Across 1.4 million sequential instructions, CiMEO achieves the highest average performance among the compared methods on five generative benchmarks spanning programming, examination-style tasks, and mathematical reasoning, while mitigating catastrophic forgetting. Subspace-overlap and principal-angle analyses further indicate that localized updates help preserve pre-trained representations, highlighting functional subnetwork localization as a practical approach to efficient and stable continual learning in LLMs.

Related

• Knowledge Interaction in Hypernetwork-based Continual Learning: Conflict and Composition
• Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
• Decoupling Memory from Context: Structured Memory for Token-Efficient Test-Time Continual Learning
• Learning from Interaction Experience via Localized Context Distillation

Source: arXiv cs.AI | 2026-10-16