Binary Rewards and Reinforcement Learning: Fundamental Challenges

arXiv:2605.02375v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for improving reasoning in language models, yet models trained with RLVR often suffer from diversity collapse: while single-sample accuracy improves, multi-sample coverage degrades, sometimes falling below the base model. We provide a structural account of this phenomenon grounded in the properties of binary rewards. Binary rewards create a fundamental degeneracy for policy gradient methods: the set of distributions maximizing expected reward is infinite, with no distinguished element. KL-control resolves this degeneracy by selecting, in the limit etao 0, the filtered model p_:=a(dotmidY_1) -- the base model conditioned on validity -- which is the unique fully valid distribution closest to the base model in KL divergence. This selection operates through a nontrivial asymmetry: the tilted distribution p_{[eta]}propto a(y),e^{v(y)/eta} converges to p_ in forward KL as etao 0, yet p_* cannot serve as a direct optimization target because KL(q,|,p_) is infinite for any full-support policy q. We develop explicit formulas relating the hyperparameter eta to the more interpretable target validity rate mu. Under model misspecification -- the typical practical regime -- the pressure to decrease eta drives the optimizer toward highly concentrated distributions over a small number of valid outputs, collapsing toward ever fewer as eta decreases, rather than toward the filtered model. We illustrate this mechanism on a toy autoregressive experiment and discuss how alternative divergences that target p_ directly -- as pursued empirically by itet{kruszewski_whatever_2026} -- avoid this failure mode by rewarding coverage of p_*'s support rather than concentration on high-validity outputs.

Source: arXiv cs.LG | 2026-05-05