Golden Handcuffs make safer AI agents

arXiv:2604.13609v1 Announce Type: new Abstract: Reinforcement learners can attain high reward through novel unintended strategies. We study a Bayesian mitigation for general environments: we expand the agent's subjective reward range to include a large negative value -L, while the true environment's rewards lie in [0,1]. After observing consistently high rewards, the Bayesian policy becomes risk-averse to novel schemes that plausibly lead to -L. We design a simple override mechanism that yields control to a safe mentor whenever the predicted value drops below a fixed threshold. We prove two properties of the resulting agent: (i) Capability: using mentor-guided exploration with vanishing frequency, the agent attains sublinear regret against its best mentor. (ii) Safety: no decidable low-complexity predicate is triggered by the optimizing policy before it is triggered by a mentor.

Robust Optimization for Mitigating Reward Hacking with Correlated Proxies
MDP Planning as Policy Inference
Representation over Routing: Overcoming Surrogate Hacking in Multi-Timescale PPO
When Can You Poison Rewards? A Tight Characterization of Reward Poisoning in Linear MDPs

Source: arXiv cs.LG | 2026-04-16

Related