The Arbitrary-Placement Problem in Entropy-Minimizing Selection, and a Residual-Entropy Formulation

arXiv:2610.05925v1 Announce Type: new Abstract: Entropy-based selection objectives suffer from a fundamental degeneracy: minimizing Shannon entropy H(p_A) rewards confident selection regardless of whether the selected candidate is informative. We address this limitation with the residual entropy D = H(p_A) - H(p_eta), where p_eta is induced by candidate trust weights. We prove the exact identity D = -KL(p_AVert p_eta) - Delta, where Delta measures whether the score-induced distribution and trust profile favor the same candidates. Boundary cases establish basic safety: under uniform trust, Dleq0 automatically, so an equal-trust, non-starving state is never penalized, while at any one-hot limit, Do0 regardless of the selected candidate. For the intermediate regime where selection occurs, we prove that Dleq0 when candidate ordering by trust agrees pairwise with ordering by informativeness, and derive a tighter certificate based on the leading candidate's margin over its competitors. These results are independent of the candidate-scoring function and apply to both stationary and dynamically changing information. Experiments with a gradient-based mixture-of-experts router confirm that the ordering conditions can hold during real optimization and show that correct ordering improves downstream performance when candidates are non-interchangeable and selections are used directly rather than averaged. Beyond routing, margin-based reweighting matches or outperforms fixed-strength baselines in a class-imbalance task, while informative selection in a production video-prediction system reduces MSE by approximately 20% and transfers to a related species. Residual entropy, therefore, provides a safety criterion for selection and a usable signal for deciding when that selection is informative.

Source: arXiv cs.LG | 2026-10-06

Related