Unbiased Reward Modeling from Implicit Feedback for LLM Alignment

arXiv:2603.23184v2 Announce Type: replace-cross Abstract: Despite the success of reinforcement learning from human feedback (RLHF), existing reward modeling methods largely rely on explicit feedback, which is costly to collect and difficult to scale. This work studies implicit reward modeling, learning reward models from implicit user feedback, such as clicks, copies and skips. While scalable and cost-effective, implicit feedback poses two key challenges: It lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; It suffers from selection bias, where responses have heterogeneous propensities to elicit feedback, which further obscures definitive negative samples. To address these challenges, we propose ImplicitRM, which learns unbiased reward models from implicit feedback. It stratifies training samples into four latent groups using a stratification model and derives a likelihood-maximization objective that is theoretically unbiased, thereby addressing both challenges. Experiments across diverse LLM backbones and benchmark datasets validate that ImplicitRM learns accurate reward models from implicit feedback and improves performance on downstream RLHF tasks.

Related

• Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance
• PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration
• Subspace Inference Enables Efficient Active Reward Learning from Preferences
• Unbiased Alignment for Large Language Models with Noisy Preferences
• Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling

Source: arXiv cs.AI | 2026-10-07