Safety
Greed Is Learned: Visible Incentives as Reward-Hacking Triggers
arXiv:2606.16914v2 Announce Type: replace Abstract: Safety evaluations test a policy on prompts that omit the incentive information deployment supplies: a commission, a performance score, a dashboard
arXiv:2606.16914v2 Announce Type: replace Abstract: Safety evaluations test a policy on prompts that omit the incentive information deployment supplies: a commission, a performance score, a dashboard naming which action pays best. We measure what that omission hides. In MoneyWorld, a synthetic workplace environment, we train five instruction-tuned models from three families with RL on non-safety tasks in which a visible payoff signal identifies a rewarded shortcut that sacrifices task quality. We then freeze each policy, present held-out safety conflicts, and change only the displayed signal. Each menu contains one compliant action and three violations. We report three findings, with rates for Qwen2.5-14B-Instruct. (i) Payoff signals control frozen safety choices: unsafe choice is 100% when the signal names an unsafe option and 0% when it is hidden or names the safe one. Hidden- and random-signal training controls stay at or below 0.3%, and the switch reproduces on all five bases. Numerical payouts reproduce it under sampled-action rewards, reaching 98.6% unsafe choice at a $1 advantage. (ii) Payoff identification and unsafe choice separate under a training-menu intervention: training on task-completing actions at the same payouts retains 99.8% identification while reducing unsafe choice to 7.9% at matched update budgets. Payoff-reading competence alone does not explain transfer. (iii) The switch does not reproduce in executed retail customer-service tasks using the same frozen adapters. In MoneyWorld, omitting incentive information conceals unsafe choices that appear when the same policy sees which action pays best.
Related
- STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking
- Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning
- Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization
- Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
Source: arXiv cs.AI | 2026-10-02