Research
WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
arXiv:2604.08558v1 Announce Type: cross Abstract: Recent decoder-only autoregressive text-to-speech (AR-TTS) models produce high-fidelity speech, but their memory and compute costs scale quadratically
arXiv:2604.08558v1 Announce Type: cross Abstract: Recent decoder-only autoregressive text-to-speech (AR-TTS) models produce high-fidelity speech, but their memory and compute costs scale quadratically with sequence length due to full self-attention. In this paper, we propose WAND, Windowed Attention and Knowledge Distillation, a framework that adapts pretrained AR-TTS models to operate with constant computational and memory complexity. WAND separates the attention mechanism into two: persistent global attention over conditioning tokens and local sliding-window attention over generated tokens. To stabilize fine-tuning, we employ a curriculum learning strategy that progressively tightens the attention window. We further utilize knowledge distillation from a full-attention teacher to recover high-fidelity synthesis quality with high data efficiency. Evaluated on three modern AR-TTS models, WAND preserves the original quality while achieving up to 66.2% KV cache memory reduction and length-invariant, near-constant per-step latency.
Related
- Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models
- Compressible Softmax-Attended Language under Incompressible Attention
- LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification
- PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing
- Dynamic sparsity in tree-structured feed-forward layers at scale
Source: arXiv cs.AI | 2026-04-13