Research
Temporal Recurrence Favors Fewer Layers
arXiv:2609.12531v1 Announce Type: new Abstract: In streaming tasks, recurrent models can carry latent computation across time, allowing each update to build on representations produced earlier. This r
arXiv:2609.12531v1 Announce Type: new Abstract: In streaming tasks, recurrent models can carry latent computation across time, allowing each update to build on representations produced earlier. This raises a basic question: once temporal recurrence provides sequential computation across steps, how much depth is still needed within each step? Prior work has shown that recurrence can make shallow models competitive. We instead study this question as a compute-allocation problem, varying within-step depth, expert width, and the number of parallel experts per layer across several compute budgets. For each budget, we compare the best observed recurrent and non-recurrent allocations and the performance they achieve under approximately matched per-step computation. Across Sokoban and autoregressive FineWeb language modeling, we find that temporal recurrence shifts the best observed compute allocation toward substantially fewer layers, with comparable or better performance.
Related
- StreamPhy: Streaming Inference of High-Dimensional Physical Dynamics via State Space Models
- Echoes of the Past: A Unified Perspective on Fading memory and Echo States
- ChronoSSM: Training for Temporally Aware Representations in Autoregressive State Space Models
Source: arXiv cs.LG | 2026-09-14