Hardware
Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72
The NVIDIA GB300 NVL72 platform set a world record by delivering 1,648 TFLOPs per GPU during the pre‑training of DeepSeek‑V3 671B, a mixture‑of‑experts (MoE) model. This achievement leveraged fifth‑ge
The NVIDIA GB300 NVL72 platform set a world record by delivering 1,648 TFLOPs per GPU during the pre‑training of DeepSeek‑V3 671B, a mixture‑of‑experts (MoE) model. This achievement leveraged fifth‑generation NVLink, 1.8 TB/s per‑GPU bandwidth, and 130 TB/s non‑blocking all‑to‑all traffic, combined with tightly integrated scale‑up and scale‑out networking (NVLink, ConnectX‑8 SuperNICs, Quantum‑X800 InfiniBand, Spectrum‑X Ethernet). Software advances such as NVIDIA Megatron Core, TorchTitan, and JAX provided 3x–10x performance gains, maintaining over 97 % throughput when scaling from 256 to 1,024 GPUs, illustrating the effectiveness of end‑to‑end hardware‑software co‑design.
Source: NVIDIA Developer | 2026-07-21