PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

arXiv:2608.30597v1 Announce Type: cross Abstract: Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as actively correcting supervision direction and strength rather than merely filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO obtains the best mean win rate against DPO (60.5 vs. 55.5 for the next-best method). Injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics further show that the routing remains stable and distinguishes flipped from weakly directional pairs.

Related

• Mitigating LLM biases toward spurious social contexts using direct preference optimization
• Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimization
• Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization
• TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generation
• AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates

Source: arXiv cs.CL | 2026-09-01