Towards Formalizing Reinforcement Learning Theory: A Robbins-Siegmund Approach
arXiv:2511.03618v2 Announce Type: replace Abstract: In this paper, we formalize the almost sure convergence of Q-learning and linear temporal difference (TD) learning with Markovian samples using the