Safety
Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards
arXiv:2610.15888v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question a
arXiv:2610.15888v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distribution itself: under a fixed rollout budget, which distribution over states minimizes the variance of the policy-gradient estimator? The answer admits a closed-form Neyman allocation, mu_{var}^pi(s) propto d^pi(s),sigma_pi(s), which connects naturally to GRPO's group-relative advantage and induces an emergent reverse curriculum as the policy improves, with no externally imposed schedule. Building on this principle, we propose VSR (Variance-driven State Resampling), a plug-in modification to GRPO using a prefix replay buffer and a zero-cost variance proxy. On mathematical reasoning benchmarks and long-horizon agentic tasks (Sokoban, BabyAI), VSR outperforms GRPO and GiGPO on most benchmarks with fewer generated tokens per update.
Related
- PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR
- Adapter Thickets: Splitting an RLVR Budget Beats Concentrating It
- Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
- Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
- Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
Source: arXiv cs.LG | 2026-10-14