Looped transformers are a popular architecture topic right now. This new technical report extends the loop across tokens. Recurrent Looped T…

Looped transformers are a popular architecture topic right now. This new technical report extends the loop across tokens. Recurrent Looped Transformer (RLT) makes the decoder recurrent over every token, prompt and response included. A causal encoder builds global KV memory. For each new token, the decoder combines the token's encoder representation with its own final hidden state from the previous token and a sliding-window cache of recent activations. With a 48-layer decoder, the computation path after t tokens runs through 48t decoder blocks, while each token still executes a fixed number of blocks. Depth grows with the sequence and per-token cost stays the same. The same state transition is used for pretraining, SFT, sampling and RL replay, and nothing resets at the prompt-response boundary. RL replay rebuilds states under the current weights instead of reusing stale rollout states. The report is a design proposal. The author states that reasoning gains, hardware speedups and RL scaling are goals that have not been measured yet. Paper: https://github.com/yifanzhang-pro/recurrent-looped-tranformer Chat with Paper: https://academy.dair.ai/papers/recurrent-looped-transformer

Related

• Thinking Deeper, Not Longer: Depth-Recurrent Transformers for Compositional Generalization [R]
• Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
• Here's a link to the article: https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and

Source: DAIR.AI (X) | 2026-09-13