Rubric-Aligned Disentangled Evaluation of Human Simultaneous Interpreting

arXiv:2609.11131v1 Announce Type: new Abstract: Human simultaneous interpreting (SI) is commonly assessed with analytic rubrics separating meaning transfer, delivery quality, and temporal synchrony, yet no automatic metric is designed for rubric-aligned segment-level SI evaluation. We construct a professionally annotated corpus of 1,101 SI segments with scores for meaning transfer (LQ), delivery quality (EXP), and perceived latency (LAT). We show that structured LLM prompting and scalar supervision collapse rubric dimensions, yielding near-zero correlation with human ratings and strong cross-dimension coupling. To isolate supervision structure under identical backbone capacity, we introduce dual regression heads on a LoRA-adapted COMET-KIWI encoder. On a held-out talk-level test set, the model achieves Pearson correlations of 0.388 (LQ) and 0.301 (EXP), improving over frozen COMET-KIWI. Given low absolute rater agreement, we interpret results relative to human consistency and target stable ranking signals for formative assessment.

Related

• Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
• LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation
• GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
• Temporal Flattening in LLM-Generated Text: Comparing Human and LLM Writing Trajectories

Source: arXiv cs.CL | 2026-09-11