Safe-Support Q-Learning: Learning without Unsafe Exploration

arXiv:2604.25379v1 Announce Type: new Abstract: Ensuring safety during reinforcement learning (RL) training is critical in real-world applications where unsafe exploration can lead to devastating outcomes. While most safe RL methods mitigate risk through constraints or penalization, they still allow exploration of unsafe states during training. In this work, we adopt a stricter safety requirement that eliminates unsafe state visitation during training. To achieve this goal, we propose a Q-learning-based safe RL framework that leverages a behavior policy supported on a safe set. Under the assumption that the induced trajectories remain within the safe set, this policy enables sufficient exploration within the safe region without requiring near-optimality. We adopt a two-stage framework in which the Q-function and policy are trained separately. Specifically, we introduce a KL-regularized Bellman target that constrains the Q-function to remain close to the behavior policy. We then derive the policy induced from the trained Q-values and propose a parametric policy extraction method to approximate the optimal policy. Our approach provides a unified framework that can be adapted to different action spaces and types of behavior policies. Experimental results demonstrate that the proposed method achieves stable learning and well-calibrated value estimates and yields safer behavior with comparable or better performance than existing baselines.

Related

• Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty
• Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?
• Optimistic Policy Learning under Pessimistic Adversaries with Regret and Violation Guarantees
• RL-STPA: Adapting System-Theoretic Hazard Analysis for Safety-Critical Reinforcement Learning
• Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback

Source: arXiv cs.LG | 2026-04-29