Calibrating Model-Based Evaluation Metrics for Summarization

arXiv:2604.17200v1 Announce Type: new Abstract: Recent advances in summary evaluation are based on model-based metrics to assess quality dimensions, such as completeness, conciseness, and faithfulness. However, these methods often require large language models, and predicted scores are frequently miscalibrated, limiting their reliability. Moreover, evaluating the average quality across different summaries for a single document typically requires access to multiple reference summaries. Here, we propose a general framework that generates individual and average proxy scores without relying on reference summaries, human annotations, or expensive model-based metrics. We also propose group isotonic regression binning (GIRB), a calibration method that adjusts the raw predictions to better align with ground-truth evaluation metrics. While we focus on continuous-value scenarios, such as summarization, the method is applicable to discrete-value tasks, such as question answering. Experiments on seven datasets demonstrate that our approach consistently outperforms existing baselines.

Related

• LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation
• Pearmut: Human Evaluation of Translation Made Trivial
• C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
• VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis
• Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

Source: arXiv cs.CL | 2026-04-21