A Regret Minimization Framework on Preference Learning in Large Language Models

arXiv:2606.09124v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has enabled progress on reasoning-intensive tasks by relying on task-specific verifiers that provide automated correctness signals. However, many realistic language tasks are difficult to equip with reliable verifiers, motivating a growing reliance on reinforcement learning from human feedback (RLHF). In this setting, we argue that a closer examination of how human feedback should be interpreted is essential. We introduce Regret-based Preference Optimization (extbf{RePO}), which reframes RLHF through extit{regret minimization} rather than reward maximization. Human preferences are often shaped by extit{prospective} anticipation of outcomes and extit{counterfactual} comparisons to alternative behaviors, rather than by immediate, outcome-independent utility. extbf{RePO} captures this structure by modeling preferences as behavior-conditioned assessments of relative suboptimality. Experiments on mathematical reasoning benchmarks and human preference datasets demonstrate consistent performance gains, indicating that extbf{RePO} is an effective and human-aligned approach for training large language models.

Source: arXiv cs.AI | 2026-06-09