Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

arXiv:2603.11137v2 Announce Type: replace-cross Abstract: On-policy distillation (OPD) is a promising approach for transferring reasoning capabilities to capacity-constrained models, yet sampled-token OPD often suffers from entropy collapse and negative transfer. We identify uniform credit allocation as a key bottleneck: existing objectives obtain dense teacher feedback on student-generated trajectories, but apply it uniformly across sampled tokens despite highly heterogeneous token-level signals. Most tokens carry redundant near-zero credit, while rare heavy-tailed negative credits can dominate updates and prematurely suppress plausible reasoning trajectories. We introduce REOPOLD (Relaxed On-Policy Distillation), a framework that relaxes this uniform assignment by controlling where teacher feedback is applied, how strongly it affects the update, and when the allocation rule changes during training. Across diverse reasoning tasks, REOPOLD improves sampled-token OPD over recent post-training baselines with up to 12x higher sample efficiency, and extends to cross-vocabulary distillation, self-distillation, and test-time scaling.

Related

• Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
• Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling
• Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation
• SeOPD: Self-Evolving LLMs via Online Policy Distillation from Self-Generated Chain-of-Thought
• Diagnosing On-Policy Self-Distillation for Reasoning Language Models

Source: arXiv cs.CL | 2026-10-28