Safety
Holdout Best-of-N: Unbiased Evaluation and Its Cost
arXiv:2610.08719v1 Announce Type: new Abstract: Reusing the scores that select a Best-of-N winner can overstate its expected reward. We study evaluation from a fixed matrix of K independent scores per
arXiv:2610.08719v1 Announce Type: new Abstract: Reusing the scores that select a Best-of-N winner can overstate its expected reward. We study evaluation from a fixed matrix of K independent scores per candidate for a policy that selects using J fresh scores. A single estimator based only on this matrix is exactly unbiased for expected judge reward under every independent, stable collection of candidate-specific score laws if and only if J<K, for every pool size Mge Nge2. At J=K-1, the selector deepens as K grows. For independent Gaussian scores with common variance and fixed Mge Nge2, the unbiased minimax risk in this regime is of order sigma^2/sqrt K, attained by Holdout; allowing bias improves the rate to sigma^2/K. For two candidates, we derive the minimum-variance unbiased estimator at known variance and the sharp asymptotic unbiased minimax constant 1/(pisqrt2), which Holdout attains without knowing the variance. The cyclic average over subsets and ties can be computed in O(MKlog M) operations. At fixed selector depth, cyclic evaluation of bounded scores has O(K^{-1}) risk uniformly in pool size. The impossibility result concerns the fixed matrix: one additional fresh winner score permits unbiased evaluation of the all-K policy.
Related
- Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence
- When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits
- Stopping and Routing LLM Judge Panels
- Post-hoc Alignment of LLM-judges to Human Judgment Distribution
Source: arXiv cs.CL | 2026-10-07