Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?

arXiv:2604.08723v1 Announce Type: cross Abstract: Preference optimization methods such as DPO and KTO are widely used for aligning language models, yet little is understood about what properties of preference data drive downstream reasoning gains. We ask: what aspects of a preference pair improve a reasoning model's performance on general reasoning tasks? We investigate two distinct notions of quality delta in preference data: generator-level delta, arising from the differences in capability between models that generate chosen and rejected reasoning traces, and sample-level delta, arising from differences in judged quality differences within an individual preference pair. To study generator-level delta, we vary the generator's scale and model family, and to study sample-level delta, we employ an LLM-as-a-judge to rate the quality of generated traces along multiple reasoning-quality dimensions. We find that increasing generator-level delta steadily improves performance on out-of-domain reasoning tasks and filtering data by sample-level delta can enable more data-efficient training. Our results suggest a twofold recipe for improving reasoning performance through preference optimization: maximize generator-level delta when constructing preference pairs and exploit sample-level delta to select the most informative training examples.

Related

• What do Language Models Learn and When? The Implicit Curriculum Hypothesis
• Learning is Forgetting: LLM Training As Lossy Compression
• Drift and selection in LLM text ecosystems
• SkillFactory: Self-Distillation For Learning Cognitive Behaviors

Source: arXiv cs.AI | 2026-04-13