CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training

arXiv:2609.00892v1 Announce Type: new Abstract: Rubric-based reinforcement learning decomposes open-ended instructions into prompt-specific, flexible rubrics, making it better suited than reinforcement learning with verifiable rewards for post-training LLMs on open-ended tasks. However, static rubrics are inevitably hacked as the policy evolves, and existing dynamic approaches introduce new problems: undirected rubric extraction, unreliable hack detection, and unbounded rubric proliferation. We propose extbf{CARE} (extbf{C}ontrastive extbf{A}nchor-based extbf{R}ubric extbf{E}volution), which grounds every rubric evolution step in a high-quality anchor response generated by a frontier model conditioned on the prompt and its rubrics. At each training step, CARE contrasts the highest-scoring rollout against the anchor, enabling two complementary mechanisms: an Adaptive branch that reactively repairs reward misspecification; and a Chase branch that proactively converts frontier-level quality gaps into sharper rubrics. Together, the two branches extbf{maintain discriminative accuracy in the high-reward region}---the precise region where reward over-optimization mostly originates. Experiments on WildChecklist-9K with Qwen2.5-7B-Base and Qwen2.5-7B-Instruct show that CARE achieves state-of-the-art performance on Arena-Hard-2.0, InfoBench, and FollowBench, and is the extbf{only} method whose win rate against GPT-4.1 anchor responses shows sustained improvement throughout 300 training steps; additional results on Llama-3.1-8B-Instruct and Qwen3-8B further indicate that CARE generalizes across model families.

Related

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning
• RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning
• Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Source: arXiv cs.AI | 2026-09-02