Bellman Residual Minimization for Control: Geometry, Stationarity, and Convergence

arXiv:2601.18840v3 Announce Type: replace Abstract: Markov decision problems are most commonly solved via dynamic programming. Another approach is Bellman residual minimization, which directly minimizes the squared Bellman residual objective function. However, compared to dynamic programming, this approach has received relatively less attention, mainly because it is often less efficient in practice and can be more difficult to extend to model-free settings such as reinforcement learning. Nonetheless, Bellman residual minimization has several advantages that make it worth investigating, such as more stable convergence with function approximation for value functions. While Bellman residual methods for policy evaluation have been widely studied, methods for policy optimization (control tasks) have been scarcely explored. In this paper, we establish foundational results for the control Bellman residual minimization for policy optimization.

Related

• Beyond the Bellman Fixed Point: Geometry and Fast Policy Identification in Value Iteration
• Faster Fixed-Point Methods for Multichain MDPs
• Fitted Q Evaluation Without Bellman Completeness via Stationary Weighting

Source: arXiv cs.LG | 2026-04-28