Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine

AI Knowledge‑Base Summary

NVIDIA’s Transformer Engine augments JAX for dropless Mixture‑of‑Experts (MoE) training, delivering a 10.4× throughput boost on NVIDIA GB200 GPUs and raising DeepSeek‑V3 performance from 103 to 1,068 TFLOPS/GPU. The engine enables exact token routing without dropping or padding through a single grouped GEMM kernel that handles variable expert counts; expert‑parallel operations are accelerated by NCCL EP’s fused dispatch/combine and token deduplication, while JAX host offloading and XLA multistreaming reduce memory bottlenecks and overlap communication across NVLink and InfiniBand. On NVIDIA GB300 NVL72 hardware the full stack sustains 97 % scaling efficiency across 1,024 GPUs when training the DeepSeek‑V3 671 B model.

Source: NVIDIA Developer | 2026-09-14