Research
What do Reward Models Memorize?
arXiv:2607.24484v1 Announce Type: cross Abstract: This paper studies what discriminatively trained reward models (RMs) memorize by measuring counterfactual memorization on two human preference dataset
arXiv:2607.24484v1 Announce Type: cross Abstract: This paper studies what discriminatively trained reward models (RMs) memorize by measuring counterfactual memorization on two human preference datasets. We show that RMs 1) misallocate memorization to easy, high margin preference pairs, 2) memorize dataset-specific shortcuts (e.g., model identity, user sampling strategy), and 3) overgeneralize simple heuristic correlates of human preference (e.g., length, compliance) when confronted with unseen preference pairs. Overall, our findings indicate that discriminative training of RMs from human preference data results in biased RMs not yet capable of judging response quality in context-dependent scenarios.
Related
- Token-weighted Direct Preference Optimization with Attention
- Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
- Reward Modeling from Natural Language Human Feedback
- CiPO: Counterfactual Unlearning for Large Reasoning Models through Iterative Preference Optimization
Source: arXiv cs.CL | 2026-07-28