Research
Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximation
arXiv:2510.01721v4 Announce Type: replace Abstract: Distributionally robust reinforcement learning seeks policies that remain effective when the deployment environment differs from the one that genera
arXiv:2510.01721v4 Announce Type: replace Abstract: Distributionally robust reinforcement learning seeks policies that remain effective when the deployment environment differs from the one that generated the training data. We study model-free robust Q-learning with hi^2 uncertainty sets and linear function approximation, using data from a single trajectory of an unknown nominal MDP. Evaluating the hi^2 robust Bellman target introduces the square root of a conditional second moment, which cannot be estimated unbiasedly from one transition, while the projected robust Bellman operator need not be contractive. We address these obstacles through a variational reformulation of the robust Bellman target and a blockwise frozen-target scheme, and establish a finite-time error bound relative to the optimal robust Q-function for every gammain(0,1). A neural-network experiment illustrates how the variational target can be used in a continuous-state nonlinear-control task.
Related
- A Robust widetilde{O}(1/sqrt{T}) Rate for Unprojected TD Learning with Linear Function Approximation
- Non-Expansive Mappings in Two-Time-Scale Stochastic Approximation: Finite-Time Analysis
- Fast and Robust Convergence Rate for TD(0) with Linear Function Approximation, Universal Learning Steps and I.I.D. Samples
Source: arXiv cs.LG | 2026-09-04