Safety
Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
arXiv:2609.14636v1 Announce Type: cross Abstract: On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however,
arXiv:2609.14636v1 Announce Type: cross Abstract: On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however, is dominated by autoregressive student rollouts and scales poorly in multi-turn agentic settings. Existing acceleration methods truncate or relocate the supervision signal according to fixed, offline budgets, despite substantial variation in teacher-signal reliability both within and across trajectories. Our empirical analysis on au^2-bench reveals a clear structure in this variation: informative supervision is concentrated in the prefix of each turn, and, most importantly for multi-turn agentic training, the cross-turn loss of teacher endorsement is temporally locked to the student's first erroneous action rather than accumulating gradually over turns. Building on these findings, we propose STRIDE (Stop-and-Restart on-policy Distillation acceleration), which combines two complementary techniques: adaptive early stopping, which terminates a rollout once the cumulative teacher log-probability falls below an out-of-distribution threshold, and a prefix buffer, which caches high-quality prefixes and restarts generation at the weakest correct turn. Together, these mechanisms induce a data-driven curriculum that progressively extends coverage to later turns. On au^2-bench retail, our method matches full-trajectory OPD and exceeds the 30B teacher at a 3.73imes speedup, surpasses the baseline itself at 2.34imes, and retains a 4.51imes speedup under cross-domain multi-teacher training. As a supplementary generalization test beyond the agentic setting, STRIDE outperforms full OPD on AIME 2025 at a 5.10imes speedup and on AIME 2024 at a 3.08imes speedup; averaged across the two evaluations, both fixed-budget truncation baselines remain below full OPD.
Related
- Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
- DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization
- CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning
- When Should Multi-Round RAG Stop? Structured Stopping Judgments and Retrieval Reduction in Search-R1
Source: arXiv cs.CL | 2026-09-15