This is one of the most useful writeups I have seen on keeping an LLM judge effective in production. (bookmark it) Netflix runs judges over …

This is one of the most useful writeups I have seen on keeping an LLM judge effective in production. (bookmark it) Netflix runs judges over hundreds of thousands of show-level recommendation explanations per week, served to millions of members on mobile. They describe the judge as a lifecycle with four phases rather than an artifact that you only validate once. > Birth defines multiple evaluation criteria and builds curated benchmarks with human labels and rationales. > Training refines the judge's rubric through Reasoning-Aligned Rubric Tuning, using a meta-judge over reasoning output as the learning signal. > Deployment puts one judge in two roles, quality gating and reflective generation. > Monitoring runs continuous human-in-the-loop alignment that detects drift and triggers re-tuning behind a review gate. A five-week A/B test over tens of millions of members shifted viewing toward previously unwatched content and increased successful browse-to-play sessions against a no-explanation control, with no quality-related takedowns. Paper: https://arxiv.org/abs/2608.18300 Track more trending AI papers in our academy: https://academy.dair.ai/

Related

• The largest LLM-as-a-Judge reliability audit yet. Researchers ran 21 judges from nine providers over roughly 541,000 judgments on MT-Bench, …
• When does combining LLMs help? Great analysis on combining language models, measured across 67 models from 21 providers. Any policy that rou…
• NEW paper from Sakana AI (ICLR 2026). A 7B Conductor model just hit SOTA on GPQA-Diamond and LiveCodeBench by orchestrating other LLMs inste…

Source: DAIR.AI (X) | 2026-08-24