The Extender: A Log-Structured Transformer

arXiv:2609.32759v1 Announce Type: new Abstract: We introduce the Extender, a log-structured variant of the standard Transformer architecture. In a standard Transformer, each layer communicates with subsequent layers exclusively via the residual mathbf{h}, a superposition channel. The Extender adds a concatenation channel mathbf{x}: each layer ell emits both a residual update elta_ell which is added to mathbf{h}, and a much smaller extension epsilon_ell which is appended to mathbf{x}. While both the FFN and mathbf{q} see mathbf{h}, the attention mathbf{kv} projections take only mathbf{x} as input. As a result, the fully extended mathbf{x} contains the complete input for the mathbf{kv} projections of all layers, reducing the persistent attention memory footprint from 2Ld_{model} to sum|epsilon_ell|. We find that with |epsilon_ell|=32, the Extender matches Transformer accuracy on short-context (CORE) tasks at 199M-924M parameters, and exceeds Transformer accuracy on long-context (RULER) workloads, again at 924M parameters. For our 1664-wide, 924M model, the Extender's persistent attention memory footprint is 104imes smaller than MHA. The memory savings grow with model width.

Source: arXiv cs.LG | 2026-09-29