Safety
Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation
arXiv:2609.16937v1 Announce Type: cross Abstract: On-policy distillation (OPD) has emerged as an effective approach for large language model post-training, yet existing objectives face a trade-off bet
arXiv:2609.16937v1 Announce Type: cross Abstract: On-policy distillation (OPD) has emerged as an effective approach for large language model post-training, yet existing objectives face a trade-off between objective fidelity and optimization stability. Token-level OPD provides stable but local supervision, whereas sequence-level OPD captures future credit at the cost of horizon-dependent variance. We establish a unified temporal-credit view of these formulations, showing that practical token-level OPD can be interpreted as a temporal approximation to the sequence-level reverse-KL gradient. Building on this connection, we propose gammaOPD, which uses discounted temporal credit assignment to balance long-horizon supervision and optimization stability, while admitting a horizon-independent variance bound. We further develop a reward-compatible bounded mixing (RBM) mechanism for gammaOPD that balances verifiable outcome feedback with the discounted OPD advantage to move beyond purely teacher-dependent optimization. Experiments on mathematical and code reasoning demonstrate consistent improvements over existing OPD methods across vanilla, size-mismatched, and multi-teacher distillation settings.
Related
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
- The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
- Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
- Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective
Source: arXiv cs.AI | 2026-09-16