Parameter-Efficient Retrievers for Polish and European Languages

arXiv:2609.12913v1 Announce Type: new Abstract: Dense retrieval systems increasingly rely on multi-billion-parameter language models, whose memory and computational requirements make large-scale indexing, frequent corpus updates, and low-latency serving costly. We present a three-stage training pipeline for developing compact and efficient retrievers that remain competitive with substantially larger models. The pipeline combines cross-lingual alignment, relational knowledge distillation, and contrastive fine-tuning. It requires no original ground-truth relevance labels, relying exclusively on supervision generated by strong embedding models and rerankers utilised as teachers. Using this pipeline, we develop PolDense and EuroDense, both supporting contexts of up to 8,192 tokens. PolDense is a family of six Polish retrievers ranging from 17M to 1B parameters. EuroDense is a 435M-parameter retriever supporting nine European languages. We conduct an extensive evaluation covering 41 Polish and 150 multilingual retrieval tasks. The results demonstrate strong quality-efficiency trade-offs. PolDense-1B outperforms the evaluated retrievers with up to 9B parameters, while the PolDense family forms the Pareto frontier across model sizes. Among the evaluated models below 1B parameters, EuroDense ranks first in both task-averaged and language-averaged performance and leads in seven of nine languages. We release all models publicly.

Related

• DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillation
• Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models
• Mnemis: Dual-Route Retrieval on Hierarchical Graphs for Long-Term LLM Memory
• When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search

Source: arXiv cs.CL | 2026-09-14