Likelihood Hacking in Probabilistic Program Synthesis

arXiv:2603.24126v2 Announce Type: replace Abstract: When language models are trained by reinforcement learning (RL) to write probabilistic programs, they can artificially inflate their marginal-likelihood reward by producing programs whose data distribution fails to normalise instead of fitting the data better. We call this failure likelihood hacking (LH). We formalise LH in a core probabilistic programming language (PPL) and give sufficient syntactic conditions for its prevention, proving that a safe language fragment L_{ext{safe}} satisfying these conditions cannot produce likelihood-hacking programs. Empirically, we show that GRPO-trained models generating PyMC code discover LH exploits within the first few training steps, driving violation rates well above the untrained-model baseline. We implement L_{ext{safe}}'s conditions as exttt{SafeStan}, a LH-resistant modification of Stan, and show empirically that it suppresses LH under optimisation pressure. These results show that language-level safety constraints are both theoretically grounded and effective in practice for automated Bayesian model discovery.

Related

• Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error
• Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
• Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning
• Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards

Source: arXiv cs.LG | 2026-08-19