Highly recommended read on looped and depth-grown models. There is a lot of interest in looped and depth-grown language models for reasoning…

Highly recommended read on looped and depth-grown models. There is a lot of interest in looped and depth-grown language models for reasoning. The question here is what reusing layers does to in-context learning. At 3B parameters, both grown and looped models gain more from later context than a dense model of the same depth. They score higher on GSM8K at every number of demonstrations and predict unseen formal languages more accurately. In controlled tasks, they pick up in-context learning earlier. When a task can be solved either from context or by memorizing fixed labels in the weights, they keep relying on context. The mechanism shows up in the attention heads. Growing depth by copying layers also copies early induction heads, and most of the strongest final heads descend from a few early ancestors. Looped models show the same repeated pattern across passes of their shared block. https://arxiv.org/abs/2610.31421 Chat with Paper: https://academy.dair.ai/papers/how-depth-growth-and-looping-shape-in-context-learning-in-language-models-2610.31421

Related

• Sparse Layers are Critical to Scaling Looped Language Models
• Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models
• When Does Sparsity Mitigate the Curse of Depth in LLMs

Source: DAIR.AI (X) | 2026-10-25