From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning

arXiv:2505.22203v3 Announce Type: replace-cross Abstract: Trustworthy verifiers are essential for the success of reinforcement learning with verifiable reward (RLVR), which is the core methodology behind various large reasoning models such as DeepSeek-R1. In complex domains like mathematical reasoning, rule-based verifiers have been widely adopted in previous works to train strong reasoning models. However, the reliability of these verifiers and their impact on the RL training process remain poorly understood. In this work, we take mathematical reasoning as a case study and conduct a comprehensive analysis of various verifiers in both static evaluation and RL training scenarios. We show that widely used rule-based verifiers fail to recognize equivalent answers in different formats, leading to substantial false negatives that increasingly hinder RL performance as the policy model gets stronger. Model-based verifiers substantially improve static accuracy but are highly susceptible to reward hacking during RL, where they misclassify certain patterns in responses as correct, particularly after fine-tuning. Our findings underscore the challenges inherent to both rule- and model-based verifiers and provide insights toward developing more accurate and robust reward systems for reinforcement learning.

Related

• Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
• Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
• RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards
• IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage

Source: arXiv cs.AI | 2026-08-28