Research
Training Time Prediction for Mixed Precision-based Distributed Training
arXiv:2604.16145v1 Announce Type: cross Abstract: Accurate prediction of training time in distributed deep learning is crucial for resource allocation, cost estimation, and job scheduling. We observe
arXiv:2604.16145v1 Announce Type: cross Abstract: Accurate prediction of training time in distributed deep learning is crucial for resource allocation, cost estimation, and job scheduling. We observe that the floating-point precision setting is a key determinant of training time, leading to training time variations of ~2.4x over its minimum. However, existing studies on distributed training time prediction rely on static model computation graphs that do not capture precision variations, including mixed precision. According to our experiments, training time prediction without considering precision results in significant prediction errors - reaching up to 147.85% in mean absolute percentage error (MAPE). To address this issue, we propose a precision-aware distributed training time predictor that achieves robust accuracy across diverse precision settings, including mixed precision, with 9.8% MAPE.
Related
- StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
- Structured Exploration and Exploitation of Label Functions for Automated Data Annotation
- Gypscie: A Cross-Platform AI Artifact Management System
- From Navigation to Refinement: Revealing the Two-Stage Nature of Flow-based Diffusion Models through Oracle Velocity
Source: arXiv cs.AI | 2026-04-20