Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression

arXiv:2511.11973v2 Announce Type: replace Abstract: Offline reinforcement learning (RL) enables policy learning from fixed datasets without further environment interaction, making it particularly valuable in high-risk or costly domains. Extreme Q-Learning (XQL) is a recent offline RL method that models Bellman errors using the Extreme Value Theorem, yielding strong empirical performance. However, XQL and its stabilized variant MXQL suffer from notable limitations: both require extensive hyperparameter tuning specific to each dataset and domain, and also exhibit instability during training. To address these issues, we proposed a principled method to estimate the temperature coefficient eta via quantile regression under mild assumptions. To further improve training stability, we introduce a value regularization technique with mild generalization, inspired by recent advances in constrained value learning. Experimental results demonstrate that the proposed algorithm achieves competitive or superior performance across a range of benchmark tasks, including D4RL and NeoRL2, while maintaining stable training dynamics and using a consistent set of hyperparameters across all datasets and domains.

Related

• Epistemic Robust Offline Reinforcement Learning
• Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL
• Tensor-Efficient High-Dimensional Q-learning
• Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
• DROP: Distributional and Regular Optimism and Pessimism for Reinforcement Learning

Source: arXiv cs.LG | 2026-04-15