Retrofitting Small Multilingual Models for Retrieval: Matching 7B Performance with 300M Parameters

arXiv:2510.14274v2 Announce Type: replace Abstract: Training effective multilingual embedding models presents unique challenges due to the diversity of languages and task objectives. Although small multilingual models (1 B) in the most prevalent use case: retrieval. This raises a critical question: Can smaller models be retrofitted specifically for retrieval tasks to enhance their performance? In this work, we investigate key factors that influence the effectiveness of multilingual embeddings, focusing on training data scale, negative sampling strategies, and data diversity. We find that while increasing the scale of training data yields initial performance gains, these improvements quickly plateau - indicating diminishing returns. Incorporating hard negatives proves essential for consistently improving retrieval accuracy. Furthermore, our analysis reveals that task diversity in the training data contributes more significantly to performance than language diversity alone. As a result, we develop a compact (approximately 300M) multilingual model that achieves retrieval performance comparable to or even surpassing current strong 7B models.

Related

• HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
• Improving End-to-End Training of Retrieval-Augmented Generation Models via Joint Stochastic Approximation
• Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit Fusion
• Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challenges

Source: arXiv cs.CL | 2026-04-23