Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models

arXiv:2606.30627v1 Announce Type: cross Abstract: Conservative offline training is widely advocated as a safe foundation for subsequent online adaptation: if a policy stays close to well-supported behaviour, the argument goes, it is less likely to exploit imperfections in a learned reward model. We challenge this intuition empirically and mechanistically. We train a Qwen3-14B policy under Direct Preference Optimisation (DPO) with three levels of conservatism (eta in {eta_{lo}, eta_{mid}, eta_{hi}} derived from empirical log-ratio percentiles), then adapt each checkpoint online against a learned reward ensemble (3,imes,Qwen3-1.7B) while measuring true performance on GSM8K exact-answer accuracy. We find that higher offline conservatism monotonically increases reward-hacking damage, measured by the Goodhart gap and its area under the curve (AUGC), with Spearman rho = 1.0 across all three conditions. Mechanistic analysis reveals a three-link causal chain: (i) high-eta DPO compresses policy entropy, (ii) Low-entropy policies generate responses with reduced diversity, concentrating in a narrow region of the reward model's training distribution (lower pairwise cosine distance), and (iii) despite this proximity, ensemble disagreement (epistemic uncertainty) increases with eta and is exploited faster during online optimisation. We further fit a power-law curve to the (eta, augc) data and identify a practical optimal conservatism level eta^{star} that balances alignment fidelity against hacking vulnerability. Our results suggest that the field needs calibrated, not maximal, conservatism.

Source: arXiv cs.AI | 2026-06-30