Tutorials
Learning Linear Regression with Low-Rank Tasks in-Context
arXiv:2510.04548v2 Announce Type: replace-cross Abstract: In-context learning (ICL) is a key building block of modern large language models, yet its theoretical mechanisms remain poorly understood. It
arXiv:2510.04548v2 Announce Type: replace-cross Abstract: In-context learning (ICL) is a key building block of modern large language models, yet its theoretical mechanisms remain poorly understood. It is particularly mysterious how ICL operates in real-world applications where tasks have a common structure. In this work, we address this problem by analyzing a linear attention model trained on low-rank regression tasks. Within this setting, we precisely characterize the distribution of predictions and the generalization error in the high-dimensional limit. Moreover, we find that statistical fluctuations in finite pre-training data induce an implicit regularization. Finally, we identify a sharp phase transition of the generalization error governed by task structure. These results provide a framework for understanding how transformers learn to learn the task structure.
Related
- Transformers for dynamical systems learn transfer operators in-context
- MapPFN: Learning Causal Perturbation Maps in Context
- Sample Complexity of Autoregressive Reasoning: Chain-of-Thought vs. End-to-End
- When Can LLMs Learn to Reason with Weak Supervision?
Source: arXiv cs.LG | 2026-04-24