Safety Training Modulates Harmful Misalignment Under On-Policy RL, But Direction Depends on Environment Design

arXiv:2604.12500v1 Announce Type: new Abstract: Specification gaming under Reinforcement Learning (RL) is known to cause LLMs to develop sycophantic, manipulative, or deceptive behavior, yet the conditions under which this occurs remain unclear. We train 11 instruction-tuned LLMs (0.5B--14B) with on-policy RL across 3 environments and find that model size acts as a safety buffer in some environments but enables greater harmful exploitation in others. Controlled ablations trace this reversal to environment-specific features such as role framing and implicit gameability cues. We further show that most safety benchmarks do not predict RL-induced misalignment, except in the case of Sycophancy scores when the exploit relies on inferring the user's preference. Finally, we find that on-policy RL preserves a safety buffer inherent in the model's own generation distribution, one that is bypassed during off-policy settings.

Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
Robust Optimization for Mitigating Reward Hacking with Correlated Proxies
When Can You Poison Rewards? A Tight Characterization of Reward Poisoning in Linear MDPs
Understanding and Improving Continuous Adversarial Training for LLMs via In-context Learning Theory
Synthetic Data for any Differentiable Target

Source: arXiv cs.LG | 2026-04-15

Related