Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training

arXiv:2609.10052v1 Announce Type: new Abstract: LLM agents for sequential decision tasks are often post-trained with trajectory-level outcome labels, but such labels provide little supervision for preserving multiple successful branches from the same decision state. We study this problem as successful strategy coverage: how broadly a model realizes distinct successful strategies under a fixed rollout budget. We present Direct Diversity Optimization (DDO), an offline post-training method that combines Divergence-Tree Collection (DTC) with the Reference-Relative Target-Odds Objective (RTO). DTC constructs state-aligned branch sets rooted at shared decision states, and RTO trains the model to match reference-relative targets over successful alternatives. DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop. It also achieves the highest recovery rate after local action replacement and higher task success and coverage than successful-only imitation and decoding-time diversification controls.

Related

• The Interplay of Harness Design and Post-Training in LLM Agents
• Backtranslation Augmented Direct Preference Optimization for Neural Machine Translation
• Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization
• Token-weighted Direct Preference Optimization with Attention

Source: arXiv cs.CL | 2026-09-10