TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generation

arXiv:2603.00025v3 Announce Type: replace Abstract: Direct Preference Optimization (DPO) is effective for offline alignment but poorly matched to ontology-driven structured prediction, where preferred and rejected JSON often differ by only a few schema-defining tokens. In this low-edit-distance regime, sequence-level DPO spreads gradient mass across non-critical serialization tokens (gradient dilution) and can reduce likelihood on rare preferred schema tokens (token erosion). To address these limitations, we first develop a confusion-aware preference-construction strategy combining expert-curated ambiguity patterns with validation-set SFT structured-error modes, producing minimally perturbed, schema-valid negatives for ontology-level decision errors. We then introduce Token-Adaptive Barrier Preference Optimization (TAB-PO), a post-SFT objective for token-critical structured generation with a confidence-gated token-level barrier that anchors under-confident schema tokens. On SciERC, with Llama/Qwen models, TAB-PO improves ontology-critical semantic-label and relational-linking metrics by 11.59% relative to SFT, wins 100% of comparisons against strongest token-level/sequence-level DPO variants, and surpasses strongest frontier baselines on these metrics by 14.71% relative while improving textual grounding.

Related

• Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization
• RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization
• AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates

Source: arXiv cs.CL | 2026-08-04