Local Ai
Online Inference in Distributional Temporal-Difference Learning
arXiv:2608.14408v1 Announce Type: cross Abstract: We study online statistical inference for functionals of the return distribution under a fixed policy. The return distribution is estimated by nonpara
arXiv:2608.14408v1 Announce Type: cross Abstract: We study online statistical inference for functionals of the return distribution under a fixed policy. The return distribution is estimated by nonparametric distributional temporal-difference learning from a single Markov trajectory. For the Polyak--Ruppert averaged estimator, we prove that its root-T error converges weakly to a centered Gaussian random element in Cramer space. We also prove that, conditionally on the observed trajectory, the root-T difference between the bootstrap and original averages converges weakly to the same Gaussian limit. These results justify bootstrap inference for smooth statistical functionals, including variance, CVaR, expected shortfall, and expectiles. For nonsmooth statistical functionals, we develop a local asymptotic theory for the estimated return CDF over T^{-1/2}-neighborhoods of finitely many thresholds, together with its bootstrap analogue. This theory allows us to conduct inference for nonsmooth statistical functionals characterized by CDF equations, including return quantiles.
Related
- Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learning
- Geometry Adaptive Counterfactual Distribution Learning with Diffusion-Guided Smoothing
- Bayesian Ego-graph Inference for Networked Multi-Agent Reinforcement Learning
Source: arXiv cs.LG | 2026-08-17