Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

arXiv:2609.08966v1 Announce Type: new Abstract: Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.

Related

• Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT
• Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks
• What is Missing from AI Post-Training AI: An Empirical Analysis
• When Mean CE Fails: Median CE Can Better Track Language Model Quality

Source: arXiv cs.AI | 2026-09-10