Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

arXiv:2608.28383v1 Announce Type: cross Abstract: Hybrid attention dominates frontier LLMs, yet Vision Transformers (ViTs) in multimodal LLMs lack a satisfactory hybrid design, with no consensus on why certain attention patterns work better. To fill this gap, we study ViT attention heads and find they differentiate into object- and background-specialist roles, a pattern most pronounced under full attention; we call this Semantic Head Specialization (SHS). We propose SHS-Index to quantify this specialization, show that it distinguishes full-attention from chunk-window ViTs, and find that it strongly tracks downstream benchmark performance. We then identify three structural factors that shape SHS---window interaction, token serialization, and local softmax allocation---and use them as design principles for hybrid attention. Guided by these factors, we design Ariadne Attention, a hybrid that matches full attention on 22 image and video tasks at 6.5x less attention compute. Our findings establish head specialization as a measurable property for diagnosing and designing principled hybrid ViT attention at the multimodal-LLM scale.

Related

• Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
• DASH: Fast Differentiable Architecture Search for Hybrid Attention in Minutes on a Single GPU
• HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation

Source: arXiv cs.CL | 2026-08-31