End-to-End Efficient RL for Linear Bellman Complete MDPs with Deterministic Transitions

arXiv:2603.23461v2 Announce Type: replace Abstract: We study reinforcement learning (RL) with linear function approximation in Markov Decision Processes (MDPs) satisfying linear Bellman completeness -- a fundamental setting where the Bellman backup of any linear value function remains linear. While statistically tractable, prior computationally efficient algorithms are either limited to small action spaces or require strong oracle assumptions over the feature space. We provide a computationally efficient algorithm for linear Bellman complete MDPs with deterministic transitions, stochastic initial states, and stochastic rewards. For finite action spaces, our algorithm is end-to-end efficient; for large or infinite action spaces, we require only a standard argmax oracle over actions. Our algorithm learns an arepsilon-optimal policy with sample and computational complexity polynomial in the horizon, feature dimension, and 1/arepsilon.

Source: arXiv cs.LG | 2026-07-01