Tool Verification for Test-Time Reinforcement Learning

arXiv:2603.02203v2 Announce Type: replace-cross Abstract: Test-time reinforcement learning (TTRL) has emerged as a promising paradigm for Recursive Self-Improving AI (RSI) by adapting Large Reasoning Models (LRMs) on unlabeled test inputs, using self-consensus rewards derived from majority voting over sampled rollouts. However, majority voting can mistake popularity for correctness: a spurious yet high-frequency unverified consensus may become a biased reward signal, causing test-time RL to reinforce frequent but wrong answers and collapse into an incorrect mode. We address this false-popular failure mode with T^3RL (Tool-Verification for Test-Time Reinforcement Learning), a verification-aware test-time RL framework. T^3RL grounds pseudo-label construction in external tool evidence. Concretely, a verifier utilizes external tool evidence (e.g., from code execution) to upweight verified rollouts during a verification-aware voting, producing more reliable pseudo-labels for training. Across various math difficulties (MATH-500, AMC, and AIME 2024) and diverse backbone families, T^3RL significantly improves over TTRL, with better performance on harder problems. T^3RL is positioned as a verified online data synthesizer, highlighting the role of tool verification in reliable online adaptation. T^3RL is extensible to verifiable domains and provides training and inference efficiency.

Related

• Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification
• Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains
• The Hidden Signal of Verifier Strictness: Controlling and Improving Step-Wise Verification via Selective Latent Steering

Source: arXiv cs.CL | 2026-08-27