Attention Sinks in Diffusion Transformers: A Causal Analysis

arXiv:2605.09313v1 Announce Type: new Abstract: Attention sinks -- tokens that receive disproportionate attention mass -- are assumed to be functionally important in autoregressive language models, but their role in diffusion transformers remains unclear. We present a causal analysis in text-to-image diffusion, dynamically identifying dominant attention recipients per timestep and suppressing them via paired, training-free interventions on the score and value paths. Across 553 GenEval prompts on Stable Diffusion~3 (with SDXL corroboration), removing these sinks does not degrade text-image alignment (CLIP-T) or preference proxies (ImageReward, HPS-v2) at k{=}1; only under stronger interventions (k!geq!10) does HPS-v2 exhibit a metric-dependent boundary, while CLIP-T remains robust throughout. The perceptual shifts induced by suppression are nonetheless sink-specific -- sim!6imes larger than equal-budget random masking -- revealing an empirical dissociation between trajectory-level perturbation and semantic alignment in diffusion transformers. footnote{Code available at https://github.com/wfz666/ICML26-attention-sink.}

Source: arXiv cs.CV | 2026-05-12