Squeezing More from Limited Data with Recursive Transformers

arXiv:2608.26973v1 Announce Type: new Abstract: Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners.

Related

• Scaling Multi-Hop Training Data via Graph-Constrained Path Selection
• Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
• ESsEN: Training Compact Discriminative Vision-Language Transformers in a Low-Resource Setting
• RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement

Source: arXiv cs.CL | 2026-08-28