GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

arXiv:2610.01511v2 Announce Type: replace Abstract: Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that may encode behavior that is useful for the preferred response. We introduce GAW-PO, a gradient-aligned token reweighting method for DPO that estimates, for each rejected token, whether penalizing it would interfere with the preferred update directions. Tokens whose gradients are strongly aligned with the preferred behavior receive a weaker negative contribution, while conflicting tokens retain a stronger penalty. Our method achieves the highest average performance among the evaluated preference-optimization methods, improving by 0.97 points over standard DPO and 0.65 points over the strongest competing baseline across 11 benchmarks spanning mathematics, reasoning, coding, and question answering. We further show that gradient-aligned weighting is substantially more robust to aggressive preference optimization: as the DPO regularization parameter eta decreases, standard DPO degrades sharply, whereas GAW-PO continues to improve. These results suggest that accounting for the interaction between rejected-token updates and preferred behavior provides an effective form of token-level credit assignment for preference optimization.

Related

• TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generation
• Mitigating LLM biases toward spurious social contexts using direct preference optimization
• StalePO: Anchored Token-Level Preference Optimization using Legacy Post-Edits in Machine Translation
• AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates

Source: arXiv cs.CL | 2026-10-05