A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning

arXiv:2609.13050v1 Announce Type: new Abstract: Regularization-based methods have become a standard approach for training Deep Reinforcement Learning policies against adversarial input perturbations. In this paper, we unify these methods by deriving new upper bounds on the performance gap between the nominal and worst-case policies. Each upper bound is expressed as an existing regularization objective plus a KL-divergence penalty between the nominal and worst-case policies, which further explains why adding a KL penalty improves robustness in practice. Building on these bounds, we formulate robust training as a constrained optimization problem, showing that existing methods correspond to the special case of a fixed Lagrange multiplier. We instead update the multiplier jointly with the policy to automatically tune the regularization weight. Finally, we conduct extensive adversarial evaluations across several continuous control tasks to validate our theoretical analysis.

Related

• Robust Policy Optimization via Adversarial Importance Sampling
• Stationary Reweighting Yields Local Convergence of Soft Fitted Q-Iteration
• Robust Driving Control for Autonomous Vehicles: An Intelligent General-sum Constrained Adversarial Reinforcement Learning Approach
• Rethinking the Divergence Regularization in LLM RL

Source: arXiv cs.LG | 2026-09-14