Learning Process Rewards via Reasoning State Propagation

arXiv:2609.39220v1 Announce Type: new Abstract: Process reward models (PRMs) have demonstrated notable effectiveness in test-time scaling and reinforcement learning by providing fine-grained signals for evaluating intermediate reasoning states, but their training relies heavily on costly process annotations. A natural way to alleviate this dependence is to complement limited process supervision with scalable outcome supervision. However, existing PRMs often model reasoning prefixes independently, providing no explicit mechanism for effectively using final outcome to guide the learning of intermediate reasoning states. We introduce Reasoning State Propagation (RSP), which represents each reasoning prefix with a binary validity state and models transitions between successive states across the reasoning trajectory. Specifically, RSP predicts a break probability that a valid state becomes invalid and a repair probability that an invalid state returns to valid. By propagating these transitions, RSP connects intermediate states to the final state, allowing process annotations to supervise intermediate states while outcome labels supervise the final state and can provide learning signals to preceding steps. Across reasoning search, response selection, and reinforcement learning, RSP consistently outperforms representative PRM baselines, with average improvements over Qwen2.5-Math-PRM of 5.6% in beam search and 2.1% in reinforcement learning.

Related

• Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guide
• GeoSolver: Scaling Test-Time Reasoning in Remote Sensing with Fine-Grained Process Supervision
• SCPRM: A Schema-aware Cumulative Process Reward Model for Knowledge Graph Question Answering
• Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards

Source: arXiv cs.AI | 2026-10-01