Multi-Stream Attention Residuals

arXiv:2610.13630v1 Announce Type: new Abstract: Attention Residuals (AttnRes) enables content-dependent retrieval over network depth, which can be viewed as single-stream softmax attention along the depth axis. Inspired by the success of multi-head attention along the sequence axis, we introduce Multi-Stream Attention Residuals (MSAR), an extension of AttnRes that generalizes this mechanism to multi-head depth attention, yielding substantial performance gains with minimal additional overhead. MSAR draws stream diversity from sequence context through multi-scale causal convolutions, allowing different heads to retrieve historical representations through distinct contextual views while preserving the integrity of the hidden representation. The retrieved streams are combined through lightweight adaptive mixing to generate the single-stream layer input. By retaining single-stream sources and expanding them on demand, MSAR supports multi-stream computation without requiring persistent multi-stream storage or communication. Across all 12 downstream benchmarks at the 18B and 28B MoE scales, MSAR consistently outperforms prior residual variants, including standard residual connections, manifold-constrained Hyper-Connections (mHC), and AttnRes. Scaling-law experiments yield an iso-loss compute speedup of 1.58imes over the standard residual baseline, compared with 1.35imes for AttnRes and 1.30imes for mHC. These results make MSAR a promising residual architecture for large-scale pre-training.

Related

• Low-Rank Attention Residuals
• Depth-Attention: Cross-Layer Value Mixing for Language Models
• LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs
• Retrieval Capacity of Self-Attention Under Competition

Source: arXiv cs.CL | 2026-10-12