Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers

arXiv:2505.04842v2 Announce Type: replace-cross Abstract: Prevalent reinforcement learning~(RL) methods for fine-tuning LLM reasoners, such as GRPO or Leave-one-out PPO, abandon the learned value function in favor of empirically estimated returns. This hinders test-time compute scaling that relies on using the value-function for verification. Yet if parallel test-time compute is already part of the deployment plan, training should be designed to support it. In this work, we propose RL^V that augments any ``value-free'' RL method by jointly training the LLM as both a reasoner and a generative verifier using RL-generated data, adding verification capabilities without significant overhead. Empirically, RL^V boosts MATH accuracy by over 20% with parallel sampling and enables 8-32imes efficient test-time compute scaling compared to the base RL method. RL^V also exhibits strong generalization capabilities for both easy-to-hard and out-of-domain tasks. Furthermore, RL^V achieves 1.2-1.6imes higher performance when jointly scaling parallel and sequential test-time compute with a long reasoning R1 model. More broadly, RL^V instantiates the principle of co-training for test-time scaling: jointly optimizing for task performance and a capability useful at inference, using data that RL training already produces.

Related

• Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning
• Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
• Synthetic POMDPs to Challenge Memory-Augmented RL: Memory Demand Structure Modeling
• When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
• From Pixels to Digital Agents: An Empirical Study on the Taxonomy and Technological Trends of Reinforcement Learning Environments

Source: arXiv cs.AI | 2026-04-14