Safety
When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR
arXiv:2605.19425v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for advanced reasoning in Large Language Models (LLMs), but rol
arXiv:2605.19425v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for advanced reasoning in Large Language Models (LLMs), but rollout samples are expensive to obtain, making sample efficiency a critical bottleneck. A natural remedy is to reuse each rollout batch for multiple gradient updates, a standard practice in classical RL. Yet in RLVR, this amplifies policy shift, leading to severe performance degradation. Detecting the onset of degradation early enough to stop reuse remains an open and challenging problem. We close this gap by identifying the extit{Disproportionate Weight Divergence (DWD)} phenomenon: performance degradation is synchronized with a sharp surge in the exttt{lm_head} weight change, while intermediate layers remain stable. Empirically, we verify that DWD emerges consistently across diverse LLMs and tasks. Theoretically, we prove that (i) harmful gradients concentrate at the exttt{lm_head} while intermediate layers are structurally attenuated, and (ii) the exttt{lm_head} gradient norm lower-bounds the policy divergence. These results establish the exttt{lm_head} gradient norm as a principled, real-time signal of catastrophic policy shift. Guided by this insight, we propose extit{Dynamic Gradient Gating (DGG)}, a lightweight intervention that monitors the exttt{lm_head} gradient norm in real time and intercepts harmful gradients before they corrupt the optimizer. DGG consistently matches or exceeds the standard single-use baseline, achieving up to 2.93imes sample efficiency and 2.14imes wall-clock speedup across math, ALFWorld, WebShop, and search-augmented QA tasks.
Source: arXiv cs.AI | 2026-05-20