Research
Localizing Transfer Between Memorization Tasks
arXiv:2610.00771v1 Announce Type: new Abstract: A central puzzle in transfer learning is why pre-training on one task can accelerate training or improve performance on another task, and what mechanism
arXiv:2610.00771v1 Announce Type: new Abstract: A central puzzle in transfer learning is why pre-training on one task can accelerate training or improve performance on another task, and what mechanisms underlie this transfer. In this work, we examine the transfer between memorization tasks of random input-output mappings. We find two surprising transfer patterns: equivalent transfer, where each additional pre-training epoch saves approximately one downstream fine-tuning epoch; and non-equivalent transfer, where pre-training on a mismatched task can be even more efficient than directly training on the downstream task itself. Through ablation experiments, we decompose and localize the transfer into two separate effects: a "trivial" magnitude-driven transfer in the last layer, and a "non-trivial" structure-driven transfer, partially attributable to the covariance of the other layers. These results advance our understanding of the underlying mechanisms of transfer learning and have the potential to lead to principled pre-training strategies.
Related
- From Order to Distribution: A Spectral Characterization of Forgetting in Continual Learning
- Matched-Learning-Rate Analysis of Attention Drift and Transfer Retention in Fine-Tuned CLIP
- SMART: A Spectral Transfer Approach to Multi-Task Learning
- Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning
- Near-optimal and Efficient First-Order Algorithm for Multi-Task Learning with Shared Linear Representation
Source: arXiv cs.LG | 2026-10-02