Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning

arXiv:2504.13818v4 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as the leading approach for enhancing reasoning capabilities in large language models. However, it faces a fundamental compute and memory asymmetry: rollout generation is embarrassingly parallel and memory-light, whereas policy updates are communication-heavy and memory-intensive. To address this, we introduce PODS (Policy Optimization with Down-Sampling), which decouples rollout generation from policy updates by training only on a strategically selected subset of rollouts, maintaining learning quality while dramatically reducing update costs. We propose a principled subset selection criterion, max-variance down-sampling, that maximizes reward diversity, and provide an efficient O(nlog n) implementation. Empirically, Group Relative Policy Optimization (GRPO) with PODS achieves the peak test accuracy of vanilla GRPO at least mathbf{1.7imes} faster across the different reasoning benchmarks and hardware configurations we tested.

Related

• Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis
• Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
• A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
• StaRPO: Stability-Augmented Reinforcement Policy Optimization
• SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
• AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin

Source: arXiv cs.AI | 2026-04-14