A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT

arXiv:2602.22014v2 Announce Type: replace Abstract: Diversity has been gaining interest in the NLP community in recent years. At the same time, state-of-the-art transformer models such as ModernBERT use very large pre-training datasets, which are driven by size rather than by diversity. This summons to investigate the impact of diversity on pre-training. We do so in this study, with the express intent of reducing pre-training dataset size, while retaining at least comparable performance. We compare diversity-driven sampling algorithms, and we use the best one to pre-train several ModernBERT models on French with a fixed compute budget. We fine-tune and evaluate them on a variety of French benchmarks. We compare them with models pre-trained on randomly sampled data of commensurate size, with the same compute budget. We find that both random and diversity-driven sampling may reduce the pre-training dataset by up to 94% and the pre-training time by up to 73% while maintaining performance. Moreover, in some tasks, the inherent quality of models, estimated via head-only fine-tuning, is up to 10 points higher with diversity sampling than with random sampling.

Related

• LegalMidm: Use-Case-Driven Legal Domain Specialization for Korean Large Language Model
• Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang Poetry
• Compositional Steering of Large Language Models with Steering Tokens
• Semantic Gradients Interactions in SSD: A Case Study in Racial Identity and Hate Speech

Source: arXiv cs.CL | 2026-10-26