Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization

arXiv:2609.39402v1 Announce Type: new Abstract: Value-model-free RLVR methods such as GRPO assign uniform advantages to all tokens in a rollout, ignoring that tokens contribute unequally. Recent methods use token entropy as an importance proxy but compute it globally across the batch, conflating importance with prompt difficulty and positional trends. We argue that importance should instead be measured relative to the local context of each token. We introduce proximal entropy, a local measure of token importance relative to neighboring tokens, and prove it is invariant to both confounders. Proximal Entropy Policy Optimization (PEPO) uses it to weight per-token advantages and outperforms GRPO and entropy-based baselines on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We also show the formulation generalizes to other algorithms where substituting proximal entropy into existing methods improves, and applying it to single-stream RL succeeds where global entropy fails.

Related

• ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate
• Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models
• Visual sensitivity is not claim retractability: persistence-aware credit assignment for multimodal reinforcement learning
• DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewards

Source: arXiv cs.AI | 2026-10-01