Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gap

arXiv:2610.15778v1 Announce Type: new Abstract: When a policy is trained with GRPO against a frozen, reference-free LLM judge, the judge accepts more of its answers, but in a post-hoc analysis we detect no gain in held-out greedy accuracy: in all 12 (policy, judge) cells, the seed-mean gain never exceeded 0.04. A gold-reward control with the same recipe raised the 1.5B policy's greedy accuracy by +0.054 [0.013, 0.096], so for this policy the flat accuracy is specific to the judge reward; for the 0.5B policy, gold reward also gave no detectable gain. In the four rising cells we checked, the early rise in training-rollout accuracy is mostly sampling sharpening, as it is under gold reward. We then ask whether the policy-judge gap or the judge sets how far and how fast RL exploits the judge's false positives. We train 0.5B and 1.5B Qwen2.5 policies against four judges at least as large (Qwen2.5-1.5B/3B/7B, Llama-3.2-3B) and the 1.5B policy against one smaller judge (Qwen3-0.6B). The depth of judge hacking (the plateau) tracks the judge: it correlates with judge AUC (Spearman -0.69 [-0.94, -0.14] over 12 cells), and within a family the larger judge is fooled less deeply. For the pace (onset), two pre-specified pairs contradict the gap account (the larger-gap cell was fooled 3.5x sooner at the pre-specified threshold). With 12 cells we cannot formally separate judge AUC from the gap for either endpoint. The smaller judge was not fooled sooner than a judge as large as the policy, though it also differs in family and template. In a blind test on three held-out judges, a hash-locked step-0 predictor did not beat step-0 AUC, but three cells cannot separate them. At 7-8B, judges from three families plateaued at 0.37, 0.67 and 0.85.

Source: arXiv cs.LG | 2026-10-14

Related