Why dynamically routing multi-timescale advantages in PPO causes policy collapse (and a simple decoupled fix) [R]
DGX agentThis Reddit post discusses a known instability in PPO when advantage estimates operating across different temporal scales (e.g., short-horizon and long-horizon returns) are dynamically routed or mixed