Research
Scaling Laws for Mixture Pretraining Under Data Constraints
Apple derives a repetition-aware mixture scaling law showing scarce target data can be reused ~15–20× when mixed with generic corpora.
Scaling Laws for Mixture Pretraining Under Data Constraints
What the paper does
Studies mixture pretraining when valuable target data (low-resource languages, specialized domains) is scarce. Across 2,000+ training runs spanning model/dataset sizes and multilingual, domain-specific, and quality-filtered mixtures, the authors introduce a repetition-aware mixture scaling law.
Key findings
- Repetition is central to target-domain performance; mixture training tolerates far more repetition than single-source training.
- Scarce target corpora can be reused roughly 15–20 times; optimal repeats depend on target size, compute, and model scale.
- Optimizing the scaling law yields principled mixture recommendations under data constraints.
Why it matters
Gives practitioners a quantitative recipe for domain/language adaptation instead of trial-and-error mixing—directly relevant to specialized local and enterprise model training.
Related
Source: https://machinelearning.apple.com/research/scaling-laws-mixture-pretraining | 2026-08-20