Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core

Megatron‑LM PR #7262 introduces a workflow that records ordered per‑rank tensor fingerprints to pinpoint and correct nondeterminism in large‑scale pretraining. By implementing low‑overhead kernel tweaks—private output slots for grouped‑GEMM writers and fixed reduction order—it achieves bitwise determinism over 800 training steps with only ~2 % performance cost on 2,432 GPUs across three workloads, while enabling reliable failure replay, loss‑spike debugging, system‑change validation, and recovery from interrupted runs. Megatron Core further guards against regressions through recipe, kernel, and module testing under diverse parallelism settings, including FP8 and FP4 precision.

Source: NVIDIA Developer | 2026-10-06