Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards

arXiv:2509.21882v2 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is a practical, scalable way to improve large language models on math, code, and other structured tasks. However, we argue that many headline RLVR gains are not yet well validated because reports often conflate policy improvement with three confounds: (i) budget mismatch between RLVR and baseline evaluation, (ii) attempt inflation and calibration drift that convert abstentions into confident answers, and (iii) data contamination in benchmarks. Using budget-matched reproductions and partial-prompt contamination probes, we find that several widely cited gaps shrink substantially or disappear once budgets, prompts, and dataset versions are matched, and contaminated sets are treated as memorization probes rather than evidence of reasoning. This does not mean that RLVR is ineffective, but it implies that current measurements often overstate capability gains and obscure reliability costs. We therefore propose a compact, tax-aware minimum standard for RLVR training and evaluation: budget-matched saturation curves with variance, calibration, and abstention tracking, one judge robustness stress test when LLM judges are used, and an explicit contamination screen. With these controls, RLVR remains effective and deployable in verifiable domains, but reasoning gains should be treated as provisional without them.

Related

• Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis
• Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward
• Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
• Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs
• EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget

Source: arXiv cs.AI | 2026-04-14