The largest LLM-as-a-Judge reliability audit yet. Researchers ran 21 judges from nine providers over roughly 541,000 judgments on MT-Bench, …

The largest LLM-as-a-Judge reliability audit yet. Researchers ran 21 judges from nine providers over roughly 541,000 judgments on MT-Bench, JudgeBench, and RewardBench. Findings: Validating a judge with exact-match agreement overstates its skill, because exact match does not correct for chance. Switching to Cohen's kappa deflates agreement by 33 to 41 points on MT-Bench, and judge rankings move by up to 14 positions across benchmarks. There is also a consistency paradox. Two production-deployed judges score above 0.95 test-retest reliability while carrying severe position bias above 0.10, so a judge can agree with itself every time and still be wrong in the same direction every time. Paper: https://arxiv.org/abs/2606.19544 Learn to build effective AI agents in our academy: https://academy.dair.ai/

Source: DAIR.AI (X) | 2026-06-22