Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
arXiv:2510.08008v2 Announce Type: replace Abstract: As the computational demands for pre-training Large Language Models (LLMs) continue to surge, the need for efficient training paradigms becomes crit