Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance

arXiv:2608.29472v1 Announce Type: cross Abstract: Knowledge distillation trains a small student model to reproduce the outputs of a large teacher model, and its progress is typically monitored through the teacher--student discrepancy. The quantity of ultimate interest, however, is the student's error with respect to the true task. We study the relation between these two objectives in a minimal three-party model, a true teacher (generative model), a teacher, and a student, all soft committee machines, in which the true teacher contains a shared latent factor that the teacher cannot represent, with mismatch strength controlled by a single scalar miss. Within an order-parameter description of online distillation, and exploiting closed-form (arcsine-type) expressions for all errors under error-function activations, we prove that the learning dynamics and the distillation error Ets are exactly invariant to miss, whereas the true error Etzs and the gap Delta=Etzs-Ets are strictly increasing in miss, with a rate that is amplified linearly by the complexity M_0 of the true teacher. Numerical phase diagrams over the plane spanned by true-teacher complexity and student capacity confirm the predicted deformation: the contours of Ets do not move while the landscape of Etzs rises systematically, and a teacher-miss regime, where mimicry succeeds but the task fails, expands with miss. The results give a quantitative warning against evaluating distillation solely through teacher-mimicry metrics and identify the gap Delta as a minimal diagnostic for distinguishing teacher-miss from capacity-limited failure.

Source: arXiv cs.AI | 2026-09-01

Related