Tempered Sequential Monte Carlo for Trajectory and Policy Optimization with Differentiable Dynamics

arXiv:2604.21456v1 Announce Type: new Abstract: We propose a sampling-based framework for finite-horizon trajectory and policy optimization under differentiable dynamics by casting controller design as inference. Specifically, we minimize a KL-regularized expected trajectory cost, which yields an optimal "Boltzmann-tilted" distribution over controller parameters that concentrates on low-cost solutions as temperature decreases. To sample efficiently from this sharp, potentially multimodal target, we introduce tempered sequential Monte Carlo (TSMC): an annealing scheme that adaptively reweights and resamples particles along a tempering path from a prior to the target distribution, while using Hamiltonian Monte Carlo rejuvenation to maintain diversity and exploit exact gradients obtained by differentiating through trajectory rollouts. For policy optimization, we extend TSMC via (i) a deterministic empirical approximation of the initial-state distribution and (ii) an extended-space construction that treats rollout randomness as auxiliary variables. Experiments across trajectory- and policy-optimization benchmarks show that TSMC is broadly applicable and compares favorably to state-of-the-art baselines.

Related

• Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation
• Beyond Importance Sampling: Rejection-Gated Policy Optimization
• Relative Entropy Pathwise Policy Optimization
• Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
• Mean Flow Policy Optimization

Source: arXiv cs.LG | 2026-04-24