AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation

arXiv:2610.14697v1 Announce Type: cross Abstract: Speech-conditioned generation maps speech signals to target audio outputs. Prior end-to-end speech-conditioned generative models achieve high fidelity in clean acoustic conditions, but their performance degrades substantially under environmental noise. While prior noise-robust approaches have achieved significant success in feature-level purification, maintaining consistent generation under clean and noisy conditioning remains a key challenge. To address this, we propose AlignAudio, a noise-robust generative framework based on dual alignment: (i) aligning the representations of clean and noisy speech to preserve semantic cues, and (ii) enforcing generation-level consistency within a flow matching paradigm to improve conditioning fidelity. Experiments on speech-to-audio (STA) and voice conversion (VC) tasks demonstrate that AlignAudio maintains performance comparable to baselines on clean speech while significantly outperforming them across various noisy conditions. Furthermore, the proposed dual-alignment strategy is applicable to both single- and multi-stream speech conditioning and can be integrated into different flow matching-based generation frameworks and post-training settings. Audio demonstrations and qualitative comparisons: href{https://alignaudio.github.io/AlignAudio}{extcolor{cyan}{extit{https://alignaudio.github.io/AlignAudio}}}.

Related

• DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis
• C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Space
• UniVoice: A Unified Model for Speech and Singing Voice Generation

Source: arXiv cs.AI | 2026-10-15