Research
Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning
arXiv:2610.14953v1 Announce Type: new Abstract: Contrastive reinforcement learning (CRL) learns goal-conditioned values by associating state--action pairs with the states subsequently reached, assumin
arXiv:2610.14953v1 Announce Type: new Abstract: Contrastive reinforcement learning (CRL) learns goal-conditioned values by associating state--action pairs with the states subsequently reached, assuming that no unobserved variable jointly affects the recorded action and its outcome. When offline demonstrations come from a demonstrator with information hidden from the learner, this assumption fails and the critic absorbs spurious action--outcome associations. We formalize this setting with confounded Markov decision processes and introduce counterfactual transitions that separate what the data identify from what they do not; minimizing over all compatible transitions yields a worst-case occupancy that lower-bounds the true one. Causal Contrastive Reinforcement Learning (CauCRL) learns a pessimistic counterfactual transition from positive and negative offline demonstrations and trains the critic on counterfactual rollouts, substantially improving success rates over standard CRL on confounded goal-reaching tasks.
Related
- Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
- Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL
- Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL
Source: arXiv cs.LG | 2026-10-14