Iwin Transformer: Hierarchical Vision Transformer using Interleaved Windows

arXiv:2507.18405v3 Announce Type: replace Abstract: Vision Transformers (ViTs) face two limitations: the rigid resolution dependency of positional embeddings, which complicates cross-resolution fine-tuning, and the quadratic complexity of attention. While Swin Transformer alleviates the latter through window attention, it suffers from fine-tuning. Following the philosophy "no token is an island," we present Iwin Transformer, a position-embedding-free hierarchical vision transformer that couples interleaved window attention with depthwise convolution inside a single block. Attention captures long-range dependencies, while convolution links local neighbors and implicitly encodes spatial position. This design not only reduces the quadratic complexity of attention but also enables two types of scalability: fine-tuning from low to high resolution and weight transfer from 2D to 3D. With window-size adjustment alone, direct 224^2{rightarrow}384^2 fine-tuning lifts Iwin-L from 86.4% to 87.4% top-1 accuracy on ImageNet-1K. Transferring an ImageNet-pretrained Iwin-T to video achieves 79.1% on Kinetics-400, outperforming Swin-T (78.8%) with 15.9% fewer FLOPs. Iwin also remains competitive on ADE20K segmentation and class-conditional image generation (FlashDiT). Overall, Iwin offers an effective approach to simultaneously tackling the complexity and scalability challenges in ViTs. Code and models are at https://github.com/cominder/Iwin-Transformer.

Related

• Positional Encodings Anchor Spatial Structure in Vision Transformers: A Geometric Perspective on Robustness
• From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology
• UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation
• Self-supervised pretraining for an iterative image size agnostic vision transformer

Source: arXiv cs.CV | 2026-08-26