Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation

arXiv:2601.07338v2 Announce Type: replace Abstract: Large Language Models (LLMs) have significantly advanced Machine Translation (MT), applying them to linguistically complex domains-such as Social Network Services, literature etc. In these scenarios, translations often require handling non-literal expressions, leading to the inaccuracy of MT metrics. To systematically investigate the reliability of MT metrics, we first curate a meta-evaluation dataset focused on non-literal translations, namely MENT. MENT encompasses four non-literal translation domains and features source sentences paired with translations from diverse MT systems, with 7,530 human-annotated scores on translation quality. Experimental results reveal the inaccuracies of traditional MT metrics and the limitations of LLM-as-a-Judge, particularly the knowledge cutoff and score inconsistency problem. To mitigate these limitations, we propose RATE, a novel agentic translation evaluation framework, centered by a reflective Core Agent that dynamically invokes specialized sub-agents. Experimental results indicate the efficacy of RATE, achieving an improvement of at least 3.2 points in combined system- and segment-level correlation with human judgments compared with current methods. Further experiments demonstrate the robustness of RATE to general-domain MT evaluation. Code and dataset are available at: https://github.com/BITHLP/RATE.

Related

• SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval
• Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
• Training-Free Test-Time Contrastive Learning for Large Language Models
• Structure-Grounded Knowledge Retrieval via Code Dependencies for Multi-Step Data Reasoning

Source: arXiv cs.CL | 2026-04-17