Safety
Instruction Quality Matters: Refining Instructions for Effective Preference Learning
arXiv:2608.26779v1 Announce Type: new Abstract: Preference learning optimizes models using response pairs, yet the informativeness of these pairs is fundamentally shaped by the instructions from which
arXiv:2608.26779v1 Announce Type: new Abstract: Preference learning optimizes models using response pairs, yet the informativeness of these pairs is fundamentally shaped by the instructions from which they are generated. We identify instruction quality as a hidden bottleneck in preference learning: low-quality or ambiguous instructions restrict the response-quality distribution, limiting strong chosen responses and weakening preference signals. Through Best- and Worst-of-N analyses, we show that instruction quality constrains both the ceiling and floor of sampled response quality. Motivated by this observation, we introduce an instruction-refinement pipeline that selects weak instructions using reward signals and revises them with rubric-guided LLM feedback, improving preference data without discarding examples. Across offline and online preference learning settings, experiments on multiple models and benchmarks show broad alignment improvements over original data and alternative data-improvement strategies. Further analyses indicate that instruction refinement raises achievable response quality and complements response-centric preference data curation. Overall, instruction quality emerges as a key factor governing how informative preference signals are formed for LLM alignment. Code is available at: https://github.com/01choco/instruction-refinement/
Related
- Instruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-Distillation
- In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models
- Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generation
Source: arXiv cs.CL | 2026-08-28