Safety
Efficient RL Training for LLMs with Experience Replay
arXiv:2604.08706v1 Announce Type: new Abstract: While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL,
arXiv:2604.08706v1 Announce Type: new Abstract: While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.
Related
- Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
- Self-Distilled RLVR
- Limits of Difficulty Scaling: Hard Samples Yield Diminishing Returns in GRPO-Tuned SLMs
- GIFT: Group-Relative Implicit Fine-Tuning Integrates GRPO with DPO and UNA
- Beyond Pessimism: Offline Learning in KL-regularized Games
Source: arXiv cs.LG | 2026-04-13