Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

arXiv:2605.30833v1 Announce Type: cross Abstract: On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, extbf{Supervision Fidelity Decay (SFD)}: as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce extbf{Lookahead Group Reward (ours{})}. Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, ours{} improves mean@8 by extbf{2.57} points over OPD for a 7B student, with gains increasing in longer-generation and reaching +extbf{4.92} points on AIME-26 at 39k tokens.

Source: arXiv cs.AI | 2026-06-01