Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner

arXiv:2609.12651v1 Announce Type: new Abstract: While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent work by Golz et al. (2025) demonstrated that the extit{distortion} -- defined as the multiplicative gap between the average user utility of the RLHF policy and the optimal average utility -- can scale exponentially with the Bradley-Terry temperature parameter eta when users have heterogeneous preferences. In this work, we present a fine-grained analysis of the distortion of RLHF with reward clipping and demonstrate that such exponential degradation is not a fundamental property of the algorithm but rather a consequence of distribution mismatch between the distribution generating preference data (mu) and the KL reference policy (pi_{ref}). To this end, we establish tight upper and lower bounds on the distortion of RLHF across multiple regimes of the KL regularization strength. We show that in a representative regime, under the Bradley-Terry model, the distortion is ilde{Theta}(eta B + eta), where B is an upper bound on the log density ratio between mu and pi_{ref}. In particular, when there is no distribution mismatch (i.e., mu = pi_{ref}), RLHF achieves the optimal distortion of O(eta) up to a constant. Our results suggest that, to reasonably maximize average utility with RLHF, it is preferable to use on-policy sampled preference data or to fine-tune before RLHF on data from a source close to mu.

Related

• Online Distributionally Robust LLM Alignment via Regression to Relative Reward
• Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
• Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
• Explaining and Preventing Alignment Collapse in Iterative RLHF
• Unbiased Alignment for Large Language Models with Noisy Preferences

Source: arXiv cs.LG | 2026-09-14