Research
Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning
arXiv:2608.25350v1 Announce Type: new Abstract: Vision-language models (VLMs) have emerged as a powerful source of supervision for reinforcement learning, enabling agents to leverage rich semantic kno
arXiv:2608.25350v1 Announce Type: new Abstract: Vision-language models (VLMs) have emerged as a powerful source of supervision for reinforcement learning, enabling agents to leverage rich semantic knowledge during training. Inspired by the success of preference-based reward learning (PbRL) in reinforcement learning from human feedback (RLHF), vision-language model generated image-based preferences provide an effective source for learning reward functions. This can be done by visually comparing two outcomes through the Bradley-Terry (BT) model. However, this pairwise formulation utilizes only two observations at a time, despite VLMs being capable of ranking multiple candidates. The Plackett-Luce (PL) formulation can shape a reward model with listwise rankings as opposed to pairwise preferences, allowing for a more suited use of a VLM based ranking. In this work, to our knowledge, we introduce the first framework that combines VLM-generated preferences with the Plackett-Luce model for reward learning. We evaluate our approach on Meta-World manipulation tasks and show that Plackett-Luce (PL) reward models can train robotic policies from VLM-generated rankings as effectively as pairwise Bradley-Terry, K-wise Bradley-Terry, and RL-VLM-F baselines. Across all environments, at least one PL ranking size (K in {3,4,5}) consistently performs with or outperforms other methods in mean success rate. Unlike pairwise methods, which are restricted to K=2, PL supports different ranking sizes and can therefore be adapted to the environment and desired feedback format. Our best PL configuration achieves an 86% mean final success rate and matches the Oracle baseline on Drawer Open. Overall, these results demonstrate that listwise VLM preference supervision is a competitive and flexible approach to reward learning for reinforcement learning.
Related
- From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
- xi-DPO: Direct Preference Optimization via Ratio Reward Margin
- Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders
- What Does Preference Learning Recover from Pairwise Comparison Data?
Source: arXiv cs.LG | 2026-08-27