RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

arXiv:2607.02584v1 Announce Type: new Abstract: In extbf{DiT-based video generation models equipped with 3D Rotary Position Embeddings (3D RoPE)}, the attention mechanism remains a primary computational bottleneck due to its quadratic complexity with respect to sequence length. While quantized extbf{FlashAttention} offers a promising path toward hardware acceleration, existing low-bit quantization methods overlook two critical challenges in this setting: extbf{1)} applying online rotation matrices -- a widely used technique for mitigating outliers in Queries (Q) and Keys (K) -- is difficult to reconcile with extbf{RoPE}; and extbf{2)} the non-negative attention matrix P = exp(QK - max(QK)) makes symmetric quantization waste half of the 4-bit dynamic range. In this work, we observe that the outlier distributions of Q and K are strongly affected by the dimensional partitioning of extbf{3D RoPE}. Based on this finding, we propose extbf{RotateAttention}, an efficient extbf{mixed-precision INT4 FlashAttention} framework tailored for extbf{DiT-based video generation models with 3D RoPE}, using selective extbf{FP16 fallback} for accuracy-sensitive attention blocks and denoising steps. RotateAttention introduces two core techniques: extbf{1) RoPE-aware Rotation}, which employs either mergeable rotation matrices that can be fused into RoPE or negligible-overhead matrices to mitigate RoPE-induced outliers in Q and K; and extbf{2) Range-optimized P Quantization}, which uses fixed scales and zero-points to fully exploit the extbf{INT4 numerical range} with minimal computational overhead. Experiments show that extbf{RotateAttention} preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68imes end-to-end speedup and 2.2imes kernel-level acceleration.

Source: arXiv cs.CV | 2026-07-07