Frequency-Uniform Log-Regret Reinforcement Learning for Diffusions

arXiv:2610.36758v1 Announce Type: cross Abstract: High-frequency control problems are often studied through time discretization, but refining the timestep elta increases the discrete horizon H=T/elta and can make standard finite-horizon regret guarantees deteriorate even when the physical horizon T is fixed. We study episodic value-based reinforcement learning for Euler discretizations of controlled diffusions. The performance is measured as the cumulative regret over K episodes relative to the optimal policy of the corresponding Euler model. Under stability and regularity conditions on value function gradients, Bellman residual derivatives, and fitted critics, we prove an O(log K) regret bound with constants uniform in elta. For dynamics with drift linear in unknown parameters and unknown constant diffusion covariance, we verify localized versions of these conditions for a constrained, model-based controlled-diffusion fitted-Q algorithm (CD-FQ) and obtain high-probability O(log Kloglog K) regret for fixed and finite T, again with constants uniform in elta.

Source: arXiv cs.LG | 2026-10-28