Relative Value Learning

arXiv:2607.21120v1 Announce Type: cross Abstract: In reinforcement learning, critics typically estimate absolute state values V(s), estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function Delta(s_i, s_j) = V(s_i) - V(s_j). We introduce a pairwise Bellman operator and prove it is a gamma-contraction with a unique fixed point equal to the true value differences, derive well-posed 1-step, n-step and lambda-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.

Source: arXiv cs.AI | 2026-07-24