Research
SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS
arXiv:2512.05126v2 Announce Type: replace-cross Abstract: Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still
arXiv:2512.05126v2 Announce Type: replace-cross Abstract: Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.
Related
- LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancement
- Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Models
- Utility-Oriented Visual Evidence Selection for Multimodal Retrieval-Augmented Generation
Source: arXiv cs.AI | 2026-09-16