Cross-lingual Matryoshka Representation Learning across Speech and Text

arXiv:2602.19991v2 Announce Type: replace Abstract: Speakers of under-represented languages face both a language barrier, as most online knowledge is in a few dominant languages, and a modality barrier, since information is largely text-based while many languages are primarily oral. We address this for French-Wolof by training the first bilingual speech-text Matryoshka embedding model, enabling efficient retrieval of French text from Wolof speech queries without relying on a costly ASR-translation pipelines. We introduce large-scale data curation pipelines and new benchmarks, compare modeling strategies, and show that modality fusion within a frozen text Matryoshka model performs best. Although trained only for retrieval, the model generalizes well to other tasks, such as speech intent detection, indicating the learning of general semantic representations. Finally, we analyze cost-accuracy trade-offs across Matryoshka dimensions and ranks, showing that information is concentrated only in a few components, suggesting potential for efficiency improvements.

Related

• OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models
• Just Use XML: Revisiting Joint Translation and Label Projection
• TMD-TTS: A Unified Tibetan Multi-Dialect Text-to-Speech Framework for U-Tsang, Amdo and Kham Speech Dataset Generation
• Scripts Through Time: A Survey of the Evolving Role of Transliteration in NLP
• Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challenges

Source: arXiv cs.CL | 2026-04-22