Safety
Influencing Humans to Conform to Preference Models for RLHF
arXiv:2501.06416v3 Announce Type: replace-cross Abstract: Designing a reinforcement learning from human feedback (RLHF) algorithm to approximate a human's unobservable reward function requires assumin
arXiv:2501.06416v3 Announce Type: replace-cross Abstract: Designing a reinforcement learning from human feedback (RLHF) algorithm to approximate a human's unobservable reward function requires assuming, implicitly or explicitly, a model of human preferences. A preference model that poorly describes how humans generate preferences risks learning a poor approximation of the human's reward function. In this paper, we conduct three human studies to asses whether one can influence the expression of real human preferences to more closely conform to a desired preference model. Importantly, our approach does not seek to alter the human's unobserved reward function. Rather, we change how humans use this reward function to generate preferences, such that they better match whatever preference model is assumed by a particular RLHF algorithm. We introduce three interventions: showing humans the quantities that underlie a preference model, which is normally unobservable information derived from the reward function; training people to follow a specific preference model; and modifying the preference elicitation question. All intervention types show significant effects, providing practical tools to improve preference data quality and the resultant alignment of the learned reward functions. Overall we establish a novel research direction in model alignment: designing interfaces and training interventions to increase human conformance with the modeling assumptions of the algorithm that will learn from their input.
Related
- What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data
- Hybrid-AIRL: Enhancing Inverse Reinforcement Learning with Supervised Expert Guidance
- ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
- Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs
- Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
Source: arXiv cs.AI | 2026-04-14