From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation
arXiv:2605.18865v1 Announce Type: cross Abstract: Self-attention serves as the core foundation of large-scale transformer pretraining, but its quadratic token interaction cost makes inference expensiv