Safety
Explanation Quality Assessment as Ranking with Listwise Rewards
arXiv:2604.24176v1 Announce Type: new Abstract: We reformulate explanation quality assessment as a ranking problem rather than a generation problem. Instead of optimizing models to produce a single 'b
arXiv:2604.24176v1 Announce Type: new Abstract: We reformulate explanation quality assessment as a ranking problem rather than a generation problem. Instead of optimizing models to produce a single "best" explanation token-by-token, we train reward models to discriminate among multiple candidate explanations and learn their relative quality. Concretely, we construct per-instance candidate sets with graded quality levels and train listwise and pairwise ranking models (ListNet, LambdaRank, RankNet) to preserve ordinal structure and avoid score compression typical of pointwise regression or binary preference objectives. We observe three findings: First, ranking losses consistently outperform regression on score separation across all domains tested. Second, the optimal ranking loss depends on data characteristics: listwise objectives excel with well-separated quality tiers, while pairwise methods are more robust to noisy natural annotations. Third, when trained on carefully curated and well-structured data, small encoder models can match models that are orders of magnitude larger, suggesting that data quality matters more than model scale. Finally, when used as rewards in policy optimization, ranking-based scores enable stable convergence in settings where regression-based rewards fail entirely. Code and data are available at: https://github.com/Tankiit/PPO_Learning_to_rank
Related
- SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
- Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis
- Scheduling Your LLM Reinforcement Learning with Reasoning Trees
- Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
Source: arXiv cs.AI | 2026-04-28