EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget

arXiv:2510.05837v2 Announce Type: replace Abstract: Balancing exploration and exploitation remains a central challenge in reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs). Current RLVR methods often overemphasize exploitation, leading to entropy collapse, diminished exploratory capacity, and ultimately limited performance gains. Although techniques that increase policy stochasticity can promote exploration, they frequently fail to escape dominant behavioral modes. This creates a self-reinforcing loop -- repeatedly sampling and rewarding dominant modes -- that further erodes exploration. We introduce Exploration-Enhanced Policy Optimization (EEPO), a framework that promotes exploration via two-stage rollouts with adaptive unlearning. In the first stage, the model generates half of the trajectories; it then undergoes a lightweight unlearning step to temporarily suppress these sampled responses, forcing the second stage to explore different regions of the output space. This sample-then-forget mechanism disrupts the self-reinforcing loop and promotes wider exploration during rollouts. Across five reasoning benchmarks, EEPO outperforms GRPO, achieving average relative gains of 24.3% on Qwen2.5-3B, 33.0% on Llama3.2-3B-Instruct, and 10.4% on Qwen3-8B-Base.

Related

• Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoning
• Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
• Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
• Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis
• SSPO: Subsentence-level Policy Optimization
• SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

Source: arXiv cs.CL | 2026-04-14