Simple-OPD: Demystifying Warm-up for On-policy Distillation

arXiv:2608.06802v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on its own rollouts with token-level supervision from teacher models, but its effectiveness can depend strongly on the warm-up stage before OPD. In this paper, we demystify warm-up for OPD from both data and training perspectives. For data, we find that effective warm-up relies on teacher-compatible chain-of-thought supervision, and that even incorrect teacher rollouts can provide comparable benefits to correct ones. This suggests that warm-up primarily transfers a teacher-compatible thinking pattern rather than merely correct answers. For training, we show that low-rank adaptation (LoRA) with a near-saturation training duration better balances in-domain adaptation and out-of-distribution generalization than full-parameter SFT. Based on these findings, we propose Simple-OPD, a plug-and-play initialization method that warms up the student on teacher-generated CoT with LoRA before OPD. Experiments across diverse settings demonstrate the effectiveness and robustness of Simple-OPD.

Related

• Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models
• OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
• Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
• TIP: Token Importance in On-Policy Distillation
• MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate

Source: arXiv cs.CL | 2026-08-10