Closed-Loop Dual-Reference Guidance for Accent Control in Cross-Lingual Voice Cloning

arXiv:2610.29351v1 Announce Type: cross Abstract: Zero-shot cross-lingual voice cloning often carries the reference speaker's accent into the target language. We propose closed-loop dual-reference guidance, which pairs the original reference with a reusable native anchor constructed by voice conversion (VC). The references initialize separate language-model (LM) states that guide a shared acoustic trajectory; each completed patch then updates both states, keeping their generated histories aligned with the emitted speech. Anchor construction runs once per speaker, and online synthesis uses a frozen TTS model without per-utterance VC. On 20 held-out French speakers, the fixed setting lambda^{!*}=0.33 reduces residual French LID mass by 73% versus direct cloning, at an ECAPA similarity cost of only 0.042; relative to anchor-only cloning, it recovers 0.155. The method achieves the lowest mean WER (0.009) and highest mean GOP (4.750) among the held-out test arms. In blinded preference tests, native-English listeners favor it over direct cloning on pronunciation naturalness, and over both anchor-only cloning and a TTSoVC cascade on speaker identity. Online RTF is approximately 26% lower than the cascade's (0.59 vs. 0.80), and the same setting reduces Spanish LID mass by 56% without re-tuning.

Related

• Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning
• MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery

Source: arXiv cs.CL | 2026-10-22