Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers

arXiv:2607.15456v2 Announce Type: replace-cross Abstract: Looped, weight-tied Transformers reduce parameters by reusing a single block, but decoding still stores a separate K/V cache for every recurrence step. We show that this loop-indexed cache is highly structured. For a fixed token, layer and head, K/V vectors trace a short low-rank trajectory across loops, while the head and layer axes remain much flatter. We introduce Looped Latent Attention (lla{}), a post-training cache codec that stores compact K and V latents and reconstructs loop-specific K/V vectors only when attention reads them. The default per-head codec compresses recurrence, while lla{}-2D also folds heads into one latent for the extreme-compression regime. The codec is initialized from the SVD of teacher activations and refined with logit and attention-output distillation. At matched cache budget, per-head lla{} outperforms head-axis MLA, cross-layer sharing, KV quantization and final-loop reuse, showing that the recurrent cache is low-rank but not safely collapsible to a single state. The same axis advantage holds on Ouro-2.6B-Thinking and transfers to Huginn-3.5B, where an SVD codec remains near-lossless to 32imes compression in decoder-independent evaluation. The cache reduction is exact. On one H200, the latent-store path increases measured Ouro-1.4B batch capacity at 4k context from 32 to 768 sequences at 21.3imes compression. Lastly, for long reasoning rollouts such as in MATH-500, on-policy refinement on student-generated prefixes raises accuracy at 4imes compression from 0.43 to 0.66 and reduces no-answer generations when compared to token-level off-policy distillation.

Related

• xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
• Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
• Latent Visual Cache for Video Reasoning

Source: arXiv cs.CL | 2026-07-27