Token Utility for On-Policy Distillation via Local Trust-Region Optimization

arXiv:2610.18883v1 Announce Type: new Abstract: Token-selection methods for on-policy distillation (OPD) prioritize student-visited positions using diverse signals, including uncertainty, teacher-student disagreement, and mismatch direction. We instead derive token importance from the distillation objective itself: how much can each position's loss decrease under an equal local KL budget on the student's predictive distribution? The answer is the utility s_t=g_t^op F_t^agger g_t, where g_t is the logit gradient and F_t is the categorical Fisher matrix. For reverse-KL it equals the student-weighted variance of the student-teacher log-probability ratio, and for forward-KL it equals the Pearson hi^2 divergence, so the appropriate utility follows from the choice of objective. On shared training checkpoints, several independently motivated criteria, including student entropy, KL disagreement, and TIP, concentrate their importance on nearly the same positions as the derived utility at a 50% selection fraction, while diverging at stricter fractions and for direction-specific criteria. We instantiate the utility as Fisher Utility Weighting (FUW), a continuous, mean-normalized token weighting. Across seven mathematical reasoning benchmarks and three Qwen3 student-teacher pairs, FUW improves over matched uniform OPD under both reverse-KL (1.35-4.0 macro points) and forward-KL (1.08-2.71 macro points), and is competitive with existing selective methods using a single closed-form score derived from the training objective.

Related

• Outcome-Confounded Local Supervision in On-Policy Distillation
• UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents
• Escaping the KL Agreement Trap in On-Policy Distillation
• Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates

Source: arXiv cs.LG | 2026-10-15