Research
On the Residual Scaling of Looped Transformers: Stability and Transferability
arXiv:2606.18524v2 Announce Type: replace Abstract: Looped (weight-tied) Transformers apply a shared residual block N times (h leftarrow h + arepsilon,f(h), same f at each step), increasing effective
arXiv:2606.18524v2 Announce Type: replace Abstract: Looped (weight-tied) Transformers apply a shared residual block N times (h leftarrow h + arepsilon,f(h), same f at each step), increasing effective depth without adding parameters. Prior depth-scaling analyses prescribe arepsilon = 1/!sqrt{L} for depth-L residual networks. We show that this is insufficient for looped architectures: weight sharing makes residual updates correlated across iterations, requiring the stronger scaling arepsilon = 1/N. For multi-layer blocks (L unique layers looped N times), we derive a factored parameterization arepsilon = lambda/(N!sqrt{L}) that separates the two sources of growth: 1/N controls the within-layer loop correlation, and 1/!sqrt{L} controls the across-layer variance. A key consequence is that the optimal learning rate depends only on the number of unique layers L, not on the loop count N, enabling direct hyperparameter transfer from small to large N without retuning. Experiments on looped Transformers confirm that 1/N scaling improves trainability and yields better loss than 1/!sqrt{N} scaling across loop counts.
Related
- Stability and Generalization in Looped Transformers
- On the Convergence of Gradient Descent on Learning Transformers with Residual Connections
- Forecasting Medium-Horizon Alzheimer's Disease Progression: Residual Gap-Aware Transformers for 24-Month CDR-SB Change from ADNI Clinical and Biomarker Histories
Source: arXiv cs.LG | 2026-09-14