Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning

arXiv:2607.08444v1 Announce Type: cross Abstract: In this paper, we study quantile-based distributional reinforcement learning from the perspective of statistical efficiency. We focus on distributional policy evaluation, whose goal is to characterize the return distribution, namely the distribution of discounted cumulative rewards under a given policy. To obtain a finite-dimensional representation of the return distribution, we consider the quantile fixed point eta_m induced by the quantile-projected distributional Bellman equation. Assuming access to a generative model, we construct an estimator eta_m^{(n)} based on an empirical Markov decision process. For a fixed number of quantiles m, we establish a non-asymptotic error bound for eta_m^{(n)} and eta_m under the supremum W_infty metric, showing that the estimation error scales as widetilde{O}(sqrt{m/n}) with respect to m and n. This implies that the quantile-based distributional policy evaluation problem can be solved with sample efficiency, achieving the optimal parametric sqrt{n} convergence rate. We derive the asymptotic distribution of the quantile parameters sqrt{n}(heta_m^{(n)}-heta_m) and characterize the semiparametric efficiency bound, which is attained by our estimator. Beyond the fixed-dimensional setting, we investigate the asymptotic regime in which the number of quantiles diverges. We characterize the limit covariance structure and show that it matches the semiparametric efficiency bound of the nonparametric model for distributional policy evaluation, showing that quantile-based estimators remain asymptotically efficient in the infinite-dimensional limit. Finally, we establish a Berry--Esseen theorem for smooth functionals sqrt{n}(eta_m^{(n)}(s)-eta_m(s))f, thereby providing a foundation for statistically valid inference on functionals of the quantile-projected return distribution.

Source: arXiv cs.LG | 2026-07-10