From Rollouts to Recipes: Self-Contained Post-Training for LLMs

arXiv:2609.01422v1 Announce Type: new Abstract: Post-training large language models usually applies a single training recipe to all samples, even though the model's own rollouts reveal different sample-level learning states. We propose Self-Routing, a behavior-conditioned post-training framework that uses rollout correctness and confidence to decide how each sample should be optimized. Depending on its behavior state, a sample is routed to GRPO, on-policy self-distillation, regularization, or skipping, allowing training to adapt without external teachers, extra annotations, or additional sampling. Experiments on mathematical reasoning across Qwen3 and Qwen3.5 backbones show that Self-Routing consistently improves over uniform GRPO, uniform OPSD, fixed mixtures, and simpler routing baselines. Further analyses show that the routing distribution changes over training and reduces unnecessary updates on low-signal or already stable samples.

Related

• Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
• Learning from the Self-future: On-policy Self-distillation for dLLMs
• OPSDL: On-Policy Self-Distillation for Long-Context Language Models
• SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
• MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

Source: arXiv cs.CL | 2026-09-02