On Generalizing Preference-based Reinforcement Learning to Multi-Objective Problems

arXiv:2607.11432v2 Announce Type: replace Abstract: We study reinforcement learning (RL) from pairwise human-generated trajectory comparisons. We generalize preference-based RL by formalizing a novel setting in which the expert can also label trajectory pairs as incomparable when neither trajectory "dominates" the other. We introduce the learning problem and the desiderata that a model should satisfy, showing that, when multiple objectives are present, no compliant model admits a negative log-likelihood that is convex in the utility difference. Then, we propose a novel Bradley-Terry-inspired rationality model that enables incomparability and infers a multi-dimensional reward function, and discuss its compliance with the desiderata. We upper bound the divergence between the true and learned comparison distributions when a dataset is available, separating the rate of approach to the optimum from a non-vanishing term due to non-convexity. Then, we study how learned parameters affect downstream guarantees, highlighting the impact of invariance on Pareto optimality.

Source: arXiv cs.LG | 2026-10-14