Beyond Isolated Heads: Multi-Overlapped-Head Self-Attention for Vision Transformers

arXiv:2410.14874v3 Announce Type: replace Abstract: Multi-Head Self-Attention (MHSA) is the cornerstone of Vision Transformers, allowing models to capture diverse feature representations by projecting tokens into independent subspaces. However, the standard MHSA strictly isolates these heads, preventing any information exchange during the attention computation itself. In this paper, we propose Multi-Overlapped-Head Self-Attention (MOHSA), a novel mechanism that replaces the hard division of attention heads with a soft, overlapping division. By allowing queries, keys, and values to partially overlap with those of adjacent heads, MOHSA fosters rich inter-head communication directly within the attention mechanism. We extensively investigate various overlap dimension scheduling strategies to identify optimal configurations. Comprehensive experiments in several Transformer architectures demonstrate that MOHSA outperforms standard MHSA on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1k datasets, offering a significant performance boost with negligible computational overhead. Code link: https://github.com/ZTX-100/MOHSA.

Related

• Representative Attention For Vision Transformers
• Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers
• SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention

Source: arXiv cs.CV | 2026-08-24