Distributional Inverse Reinforcement Learning

arXiv:2510.03013v3 Announce Type: replace Abstract: We propose a distributional framework for offline Inverse Reinforcement Learning (IRL) that jointly models uncertainty over reward functions and full distributions of returns. Unlike conventional IRL approaches that recover a deterministic reward estimate or match only expected returns, our method captures richer structure in expert behavior, particularly in learning the reward distribution, by minimizing first-order stochastic dominance (FSD) violations and thus integrating distortion risk measures (DRMs) into policy learning, enabling the recovery of both reward distributions and distribution-aware policies. This formulation is well-suited for behavior analysis and risk-aware imitation learning. Theoretical analysis shows that the algorithm converges with O(arepsilon^{-2}) iteration complexity. Empirical results on synthetic benchmarks, real-world neurobehavioral data, and MuJoCo control tasks demonstrate that our method recovers expressive reward representations and achieves state-of-the-art performance.

Related

• PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
• Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebysheff Scalarization
• DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty
• Best Policy Learning from Trajectory Preference Feedback
• Robust Optimization for Mitigating Reward Hacking with Correlated Proxies

Source: arXiv cs.LG | 2026-04-23