Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
DGX agentarXiv:2609.01532v1 Announce Type: new Abstract: Logit-based knowledge distillation (KD) is used to train smaller language models (LMs) via supervision from stronger teachers, but whether its benefits