Tutorials
Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity
arXiv:2407.04291v4 Announce Type: replace-cross Abstract: Modeling speech variation is key to natural, expressive generation. Speaker embeddings are commonly used to condition personalized speech syst
arXiv:2407.04291v4 Announce Type: replace-cross Abstract: Modeling speech variation is key to natural, expressive generation. Speaker embeddings are commonly used to condition personalized speech systems, but they are typically trained for speaker recognition, where intra-speaker variability is suppressed and inter-speaker separation is maximized. This objective leads to overly compact representations that may discard variations crucial for generation. We revisit this design choice and propose a sub-center modeling framework for speaker embeddings. Instead of a single prototype per speaker, we learn multiple sub-centers during discriminative training, allowing utterances to align with different prototypes. This strategy preserves structured intra-speaker variability while maintaining discriminability. In zero-shot voice conversion, our method improves intelligibility, increases pitch variability, achieves higher naturalness ratings, and retains strong speaker verification performance.
Related
- AaSP: Aliasing-aware Self-Supervised Pre-Training for Audio Spectrogram Transformers
- Parallel Complex Diffusion for Scalable Time Series Generation
- BAT: Better Audio Transformer Guided by Convex Gated Probing
- Geometry-Preserving Encoder/Decoder in Latent Generative Models
Source: arXiv cs.LG | 2026-08-31