Scaling Laws for Mixture Pretraining Under Data Constraints
Apple derives a repetition-aware mixture scaling law showing scarce target data can be reused ~15–20× when mixed with generic corpora.
Knowledge catalogue
Apple derives a repetition-aware mixture scaling law showing scarce target data can be reused ~15–20× when mixed with generic corpora.
This paper was accepted at the Workshop on Navigating and Addressing Data Problems for Foundation Models (NADPFM) at ICLR 2026. Principled domain reweighting can substantially improve sample efficienc