Issues with Value-Based Multi-objective Reinforcement Learning: Value Function Interference and Overestimation Sensitivity

arXiv:2402.06266v2 Announce Type: replace Abstract: Multi-objective reinforcement learning (MORL) algorithms extend conventional reinforcement learning (RL) to the more general case of problems with multiple, conflicting objectives, represented by vector-valued rewards. Widely-used scalar RL methods such as Q-learning can be modified to handle multiple objectives by (1) learning vector-valued value functions, and (2) performing action selection using a scalarisation or ordering operator which reflects the user's preferences with respect to the different objectives. This paper investigates two previously unreported issues which can hinder the performance of value-based MORL algorithms when applied in conjunction with a non-linear utility function -- value function interference, and sensitivity to overestimation. We illustrate the nature of these phenomena on simple multi-objective MDPs using a tabular implementation of multiobjective Q-learning.

Related

• Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation
• Replicable Reinforcement Learning with Linear Function Approximation
• Distributionally Robust Regret Optimal Control Under Moment-Based Ambiguity Sets
• Self-Aligned Reward: Towards Effective and Efficient Reasoners

Source: arXiv cs.LG | 2026-04-23