MDP Planning as Policy Inference

arXiv:2602.17375v2 Announce Type: replace Abstract: We cast episodic Markov decision process (MDP) planning as Bayesian inference over policies. A policy is treated as the latent variable and is assigned an unnormalized probability of optimality that is monotone in its expected return, yielding a posterior distribution whose modes coincide with return-maximizing solutions while posterior dispersion represents uncertainty over optimal behavior. To approximate this posterior in discrete domains, we adapt variational sequential Monte Carlo (VSMC) to inference over deterministic policies under stochastic dynamics, introducing a sweep that enforces policy consistency across revisited states and couples transition randomness across particles to avoid confounding from simulator noise. Acting is performed by posterior predictive sampling, which induces a stochastic control policy through a Thompson-sampling interpretation rather than entropy regularization. Across grid worlds, Blackjack, Triangle Tireworld, and Academic Advising, we analyze the structure of inferred policy distributions and compare the resulting behavior to discrete Soft Actor-Critic, highlighting qualitative and statistical differences that arise from policy-level uncertainty.

Related

• Multi-agent Reach-avoid MDP via Potential Games and Low-rank Policy Structure
• MDP modeling for multi-stage stochastic programs
• Beyond Pessimism: Offline Learning in KL-regularized Games
• Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling
• Relative Entropy Pathwise Policy Optimization

Source: arXiv cs.LG | 2026-04-14