Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation

arXiv:2608.18787v1 Announce Type: new Abstract: Learning robotic policies requires dense rewards that remain informative when behavior departs from successful demonstrations. Progress-based rewards estimate how far an observation has advanced along a nominal successful trajectory, but may remain high after an incorrect transition. We introduce Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations. Given the visual history up to a transition start, the model predicts the latent displacement associated with successful execution and scores the observed displacement through signed directional and symmetric magnitude agreement. This transition-level comparison penalizes wrong-direction, overshooting, and stagnant motion even when the resulting observation appears to show progress. Dream2Reward requires no failure annotations, progress labels, or synthetic negatives, and produces a dense causal reward. Across mechanism diagnostics and shared-trajectory evaluations, it provides stronger success-failure separation and more informative feedback on low-quality behavior than progress-based alternatives. Across online and offline policy learning, the same frozen reward model reduces reward hacking and supports stronger downstream performance, including in real-robot manipulation. These results show that comparing realized motion with predicted successful change provides an effective way to convert positive demonstrations into dense rewards for robot learning.

Related

• DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation
• GORDON: Graph-based Object-centric Rewards for Decomposition of Long-Horizon Manipulation
• WARP-RM: A Warp-Augmented Relative Progress Reward Model for Data Curation

Source: arXiv cs.RO | 2026-08-20