Safety
Optimistic Policy Learning under Pessimistic Adversaries with Regret and Violation Guarantees
arXiv:2604.14243v1 Announce Type: new Abstract: Real-world decision-making systems operate in environments where state transitions depend not only on the agent's actions, but also on extbf{exogenous f
arXiv:2604.14243v1 Announce Type: new Abstract: Real-world decision-making systems operate in environments where state transitions depend not only on the agent's actions, but also on extbf{exogenous factors outside its control}--competing agents, environmental disturbances, or strategic adversaries--formally, s_{h+1} = f(s_h, a_h, ar{a}_h)+omega_h where ar{a}_h is the adversary/external action, a_h is the agent's action, and omega_h is an additive noise. Ignoring such factors can yield policies that are optimal in isolation but extbf{fail catastrophically in deployment}, particularly when safety constraints must be satisfied. Standard Constrained MDP formulations assume the agent is the sole driver of state evolution, an assumption that breaks down in safety-critical settings. Existing robust RL approaches address this via distributional robustness over transition kernels, but do not explicitly model the extbf{strategic interaction} between agent and exogenous factor, and rely on strong assumptions about divergence from a known nominal model. We model the exogenous factor as an extbf{adversarial policy} ar{pi} that co-determines state transitions, and ask how an agent can remain both optimal and safe against such an adversary. To the best of our knowledge, this is the first work to study safety-constrained RL under explicit adversarial dynamics. We propose extbf{Robust Hallucinated Constrained Upper-Confidence RL} (exttt{RHC-UCRL}), a model-based algorithm that maintains optimism over both agent and adversary policies, explicitly separating epistemic from aleatoric uncertainty. exttt{RHC-UCRL} achieves sub-linear regret and constraint violation guarantees.
Related
- Beyond Pessimism: Offline Learning in KL-regularized Games
- Conformal Policy Control
- Offline-Online Reinforcement Learning for Linear Mixture MDPs
- Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?
Source: arXiv cs.LG | 2026-04-17