Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning

arXiv:2608.02034v1 Announce Type: new Abstract: Multi-step returns accelerate reward propagation in off-policy reinforcement learning, but couple the evaluation of each decision to the suboptimal logged actions that follow it, inducing a pessimistic bias that grows with the horizon. We propose Expectile n-step Q-learning (ENQ), which replaces the symmetric n-step temporal-difference (TD) loss with an asymmetric expectile loss on the action-value error, with expectile level au as the only method-specific hyperparameter added beyond n-step TD. We prove that the ENQ operator is a gamma^{n}-contraction. Under deterministic dynamics, at au=1, its bias vanishes at the optimal action-value function Q^* on covered in-support pairs, and the corresponding fixed point satisfies the separation-n instance and its multiples of the lower-bound inequality used by Long-Horizon Q-learning (LQL). Under stochastic dynamics, the operator bias admits two-sided bounds with horizon-independent noise constants. Using a single expectile level au=0.8 and a fixed backup horizon across 27 manipulation and navigation task instances, ENQ is competitive with LQL on aggregate, achieves higher measured training-step throughput in our profiling study, and benefits more from a ten-critic ensemble in a controlled scaling experiment.

Source: arXiv cs.LG | 2026-08-04