Safety
Why dynamically routing multi-timescale advantages in PPO causes policy collapse (and a simple decoupled fix) [R]
This Reddit post discusses a known instability in PPO when advantage estimates operating across different temporal scales (e.g., short-horizon and long-horizon returns) are dynamically routed or mixed
This Reddit post discusses a known instability in PPO when advantage estimates operating across different temporal scales (e.g., short-horizon and long-horizon returns) are dynamically routed or mixed together, which causes conflicting gradient signals that destabilize training and lead to policy collapse. The proposed fix decouples the multi-timescale advantage streams—keeping them separate during optimization rather than jointly routing them—so that each timescale's signal updates the policy independently without destructive interference. This approach draws on a broader trend in RL research of decoupling tightly coupled components in PPO to improve training stability, as seen in related work such as VC-PPO's decoupled GAE and DVPO's decoupled value estimation.
Related
- Representation over Routing: Overcoming Surrogate Hacking in Multi-Timescale PPO
- SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
- RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
- Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPO
- OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks
Source: r/MachineLearning | 2026-04-16