Toward Optimal Regret in Adversarial MDPs with Stochastic Hard Constraints

arXiv:2610.12153v1 Announce Type: new Abstract: We study episodic constrained Markov decision processes with adversarial losses under stochastic hard constraints. Specifically, starting from a known strictly feasible policy with margin d, we seek to obtain optimal regret while satisfying the expected cost constraints in every episode. In this setting, Stradi et al. (2025) show that a carefully designed mixing rule attains regret of order widetilde{O}(sqrt{T}/min{d,d^2}). Interestingly, they also provide a lower bound of order Omega(sqrt{T}/rho) for the same setting, where rho is the Slater margin of the offline problem and can be much larger than d. In this work, we build on their approach to obtain optimal regret dependence on these margins. Specifically, we propose MA-OPS, an algorithm that combines an optimistic search for the Slater margin with a pessimistic evaluation of the selected policies to safely learn a policy with a large feasibility margin. This policy is then used to minimize regret while satisfying the constraints at every episode. In particular, we show that MA-OPS attains regret widetilde{O}(sqrt{T}/rho + 1/(drho)). Finally, we provide a matching lower bound, showing that the dependence on T, d, rho in the regret bound is optimal up to logarithmic factors.

Source: arXiv cs.LG | 2026-10-09