Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models

arXiv:2609.08390v1 Announce Type: cross Abstract: Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.

Related

• Liberating LLM Capabilities in Full-Duplex Speech Models
• LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancement
• UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement
• Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition

Source: arXiv cs.AI | 2026-09-10