P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist

arXiv:2601.02986v2 Announce Type: replace Abstract: Recent approaches in personalized reward modeling have primarily focused on leveraging user interaction history to align model judgments with individual preferences. However, existing approaches largely treat user context as a static or implicit conditioning signal, failing to capture the dynamic and multi-faceted nature of human judgment. In this paper, we propose P-Check, a novel personalized reward modeling framework, designed to train a plug-and-play checklist generator that synthesizes dynamic evaluation criteria for guiding the reward prediction. To better align these checklists with personalized nuances, we introduce Preference-Contrastive Criterion Weighting, a training strategy that assigns saliency scores to criteria based on their discriminative power for personalized judgment. We conduct extensive experiments and demonstrate that P-Check not only improves reward accuracy but also enhances downstream personalized generation, and remains robust in OOD scenarios.

Related

• Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models
• C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
• Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty
• Reward Modeling for Scientific Writing Evaluation

Source: arXiv cs.CL | 2026-04-21