Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

arXiv:2604.12911v1 Announce Type: cross Abstract: Multilingual benchmarks guide the development of frontier models. Yet multilingual evaluations reported by frontier models are structured similar to popular reasoning and knowledge benchmarks, but across many languages. We show such benchmarks, and consequently multilingual evaluations, measure mathematical reasoning and factual recall, not multilingual proficiency. For example, thinking variants dramatically outperform instruct variants on these benchmarks, yet often perform worse on real-world multilingual tasks, such as LMArena. We propose a simple alternative: evaluate multilingual capability via round-trip translation. Given text in a source language, translate it to a target language and back; semantic gaps between the original and result expose failures in multilingual generation capabilities. Round-trip translation correlates almost perfectly (r{ho} = 0.94) with user ratings on LMArena with our benchmark, requires no human reference translations, and does not require a more capable multilingual judge than tested models. Lastly, we introduce Lost in Translation (LiT), a challenging round-trip translation benchmark spanning widely spoken languages worldwide, for realistic evaluation of multilingual frontier models.

Related

• MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation
• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
• Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models

Source: arXiv cs.AI | 2026-04-15