Safety
Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization
arXiv:2604.11510v1 Announce Type: cross Abstract: To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Spli
arXiv:2604.11510v1 Announce Type: cross Abstract: To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Split, a novel paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt. While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives. Specifically, the normal mode optimizes for task correctness, while the high-entropy mode incorporates a preference for exploration, and the two modes learn collaboratively. Extensive experiments demonstrate that our approach consistently outperforms established entropy-guided RL baselines across various model sizes in general and creative tasks. Further analysis reveals that Policy Split facilitates dual-mode exploration, where the high-entropy mode generates distinct behavioral patterns to the normal mode, providing unique learning signals.
Related
- Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis
- A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
- StaRPO: Stability-Augmented Reinforcement Policy Optimization
- SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
- The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping
Source: arXiv cs.AI | 2026-04-14