In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model

arXiv:2610.21901v1 Announce Type: cross Abstract: Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks. The prevailing strategy determines how the model behaves beyond its training distribution, controlling potential capabilities and risks. Recent work has characterised many different strategies theoretically, and empirically described the transition between them as the quality and quantity of pretraining data vary. However, there is no learning theory for this competition, hence no fundamental understanding of extit{why} one solution might be favoured over another. Here we provide such a theory in a solvable topic model---documents drawn from a finite mixture of latent topics, mirroring the heterogeneity of modern pretraining data. The dataset comes with two distinct solutions: a specialised predictor, that stores the finite topic mixture, and an in-context one, optimal for topics never seen during training. Our theory predicts their emergence at distinct sample complexities: the in-context predictor is available early, while the specialised one emerges only after the training tokens outnumber the parameters of the data-generating process. Trained transformers reproduce both scales and the separating crossover in remarkable agreement with the theory, with no fitted parameters.

Related

• Mechanism of Task-oriented Information Removal in In-context Learning
• From Relation-Specific Heads to Shared Interventions: Contextual Entrainment in Language Models
• Context Training with Active Information Seeking
• Automating Categorization of Scientific Texts with In-Context Learning and Prompt-Chaining in Large Language Models
• Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO

Source: arXiv cs.AI | 2026-10-16