Fitted Q Evaluation Without Bellman Completeness via Stationary Weighting

arXiv:2512.23805v2 Announce Type: replace-cross Abstract: Fitted Q-evaluation (FQE) is a foundational method for off-policy evaluation in reinforcement learning, but existing theory typically relies on Bellman completeness of the function class, a condition often violated in practice. This reliance is due to a fundamental norm mismatch: the Bellman operator is gamma-contractive in the L^2 norm induced by the target policy's stationary distribution, whereas standard FQE fits Bellman regressions under the behavior distribution. To resolve this mismatch, we reweight each Bellman regression step by an estimate of the stationary density ratio, inspired by emphatic weighting in temporal-difference learning. This makes the update behave as if it were performed under the target stationary distribution, restoring contraction without Bellman completeness while preserving the simplicity of regression-based evaluation. Illustrative experiments, including Baird's classical counterexample, show that stationary weighting can stabilize FQE under off-policy sampling.

Related

• Soft Q(lambda): A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces
• Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation
• Continuous-time reinforcement learning: ellipticity enables model-free value function approximation
• CROP: Conservative Reward for Model-based Offline Policy Optimization

Source: arXiv cs.LG | 2026-04-22