A Comprehensive Study of Content Representations for Speech Synthesis

arXiv:2609.30975v1 Announce Type: cross Abstract: Speech content representations are central to voice conversion, speech-to-speech translation, and multimodal language models, yet they are rarely compared under a common generative framework that directly measures what each representation contains. We address this by training a generative model conditioned solely on each representation and evaluating the generated audio along the content, speaker identity, and prosody axes. Across SSL features, supervised tokens, posteriorgrams, and neural audio codecs, we find two distinct regimes: representations that nearly reconstruct the original audio, and representations that effectively disentangle speaker identity. These results show that disentanglement depends not on supervision alone, but on the interaction between the training objective and the representation's information capacity: supervised representations only disentangle speaker identity when their capacity is sufficiently constrained.

Related

• Reconstructing Content via Collaborative Attention to Improve Multimodal Embedding Quality
• LatentVerse: A Framework for Understanding Shared and Modality-Specific Information in Multimodal Latent Representations
• Interleaved Speech Language Models Latently Work In Text
• Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models

Source: arXiv cs.LG | 2026-09-28