What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective

arXiv:2604.25132v1 Announce Type: new Abstract: Instruction-tuning datasets often contain substantial redundancy and low-quality samples, necessitating effective data selection methods. We propose an instruction data selection framework based on weighted in-context influence (wICI), which measures how effectively each candidate example reduces instruction-following difficulty for semantically related peers. Through systematic experiments, we address three key questions: what constitutes effective instruction tuning data from an in-context perspective, whether sample difficulty correlates with in-context influence, and how in-context influence translates to instruction tuning effectiveness. Experiments across multiple models and benchmarks demonstrate that our method consistently outperforms existing baselines under constrained data budgets, while empirically showing that sample difficulty negatively correlates with in-context influence.

Instruction Data Selection via Answer Divergence
A Survey on Split Learning for LLM Fine-Tuning: Models, Systems, and Privacy Optimizations
Weighting What Matters: Boosting Sample Efficiency in Medical Report Generation via Token Reweighting
Rethinking Data Mixing from the Perspective of Large Language Models

Source: arXiv cs.CL | 2026-04-29

Related