Non-Asymptotic Convergence of Stochastic Iterative Algorithms: A Lyapunov Framework

arXiv:2605.31309v1 Announce Type: new Abstract: We survey Lyapunov-based techniques for the finite-time analysis of stochastic iterative algorithms, also known as stochastic approximation (SA) algorithms, for solving fixed-point equations ar{F}(x)=x, where the operator ar{F}(dot) can only be accessed through a noisy oracle. We first focus on the standard setting in which ar{F}(dot) is contractive with respect to some norm and the noise is i.i.d., and explain how generalized Moreau envelopes serve as universal Lyapunov functions, regardless of the underlying norm. We then show how this framework yields mean-square convergence guarantees and applies to stochastic gradient descent, linear SA, and value-based reinforcement learning algorithms such as Q-learning and temporal-difference learning. Finally, we discuss extensions to Markovian noise, seminorm-contractive operators, dissipative operators, and high-probability bounds, and conclude with open problems. The goal is to present a unified and self-contained roadmap for the finite-time analysis of SA and its applications, especially in reinforcement learning.

Source: arXiv cs.LG | 2026-06-01