Very cool paper from Microsoft. The idea is to train agents on replayed teacher trajectories instead of live environment rollouts. On-policy…

Very cool paper from Microsoft. The idea is to train agents on replayed teacher trajectories instead of live environment rollouts. On-policy distillation for agentic tasks is expensive because every update needs fresh student rollouts through the environment plus teacher queries at each visited history. New research from Microsoft Research and the University of Amsterdam introduces ReOPD, which reuses pre-collected teacher trajectories as replayed prefixes. The student acts at selected steps while the teacher supplies dense per-step supervision without executing anything new. The paper also names a real pathology in multi-turn distillation, the prefix trap. Pushing histories toward the student's own distribution makes them more relevant to the student and simultaneously drags the teacher onto states where its targets are unreliable. Two-sided distribution shift between student occupancy and teacher reliability. ReOPD treats this as reliability-aware prefix distribution design and implements it with a step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across math reasoning with Python and search environments, over multiple teacher and student scales, it preserves or improves accuracy, uses zero tool calls during student training, and runs at least 4x faster per rollout. Paper: https://arxiv.org/abs/2607.04763 Learn to build effective AI agents in our academy: https://academy.dair.ai/

Related

• Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts
• Multi-Turn On-Policy Distillation with Prefix Replay
• From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation
• H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation

Source: DAIR.AI (X) | 2026-07-27