Research
Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens
arXiv:2607.26350v1 Announce Type: cross Abstract: Neural audio codecs (NACs) have become popular for obtaining speech representations as discrete tokens. Beyond compression, discrete tokens can be use
arXiv:2607.26350v1 Announce Type: cross Abstract: Neural audio codecs (NACs) have become popular for obtaining speech representations as discrete tokens. Beyond compression, discrete tokens can be used to train self-supervised learning (SSL) models. Such models, referred to as codec-based SSL models, reduce data storage and computational cost, enabling scalable SSL pre-training. However, their language sensitivity remains unclear. When the language changes, codec-based SSL models may require retraining, which undermines their efficiency. In this paper, we present a systematic analysis of language sensitivity by varying either the NAC training language or the SSL pre-training language while keeping the other fixed. Experimental results show that downstream performance is insensitive to the NAC training language but strongly dependent on the SSL pre-training language. These findings suggest that a single NAC can be reused across languages, while aligning the SSL pre-training language with the target language is crucial.
Related
- Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yoruba
- CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding
- Exploring Token-Space Manipulation in Latent Audio Tokenizers
- HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models
- Streaming Neural Speech Codecs through Time-Invariant Representations
Source: arXiv cs.CL | 2026-07-30