Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation

arXiv:2608.25089v1 Announce Type: new Abstract: Crosslingual evaluation of language models that enables fair comparisons remains a fundamental challenge in multilingual NLP. Existing studies adopt a variety of downstream tasks and intrinsic metrics with different theoretical justifications, yet there has been little empirical investigation into whether these approaches yield meaningful crosslingual conclusions. We systematically examine crosslingual evaluation approaches using controlled monolingual language models trained on parallel data with varying tokenizer vocabulary sizes and model sizes, and further validate our findings on multilingual LLMs. We further discuss challenges in achieving comparable downstream evaluation across languages. Our results show that several widely used normalized metrics introduce crosslinguistic biases rooted in tokenization, encoding, and orthographic differences. In contrast, sentence-level negative log-likelihood computed over semantically equivalent sequences provides more meaningful and consistent crosslingual comparisons.

Related

• (Towards) Scalable Reliable Automated Evaluation with Large Language Models
• When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs
• Lessons from the Trenches on Reproducible Evaluation of Language Models

Source: arXiv cs.CL | 2026-08-27