Hardware
Efficient MoE Training for Biological Foundation Models
Mixture‑of‑Experts (MoE) models scale capacity more efficiently than dense transformers by activating only a subset of expert subnetworks per token. NVIDIA’s Transformer Engine optimizes MoE training
Mixture‑of‑Experts (MoE) models scale capacity more efficiently than dense transformers by activating only a subset of expert subnetworks per token. NVIDIA’s Transformer Engine optimizes MoE training with GroupedLinear “kernel‑sharding” to reduce GPU kernel launch overhead, MXFP8 8‑bit precision to cut memory usage, and a fused Sequential API that combines GroupedLinear, ScaledSwiGLU, and routing‑weight scaling into a single CUDA kernel. In an eight‑GPU benchmark on NVIDIA B200 Tensor‑Core GPUs, the BioNeMo recipe achieved up to 2.21× higher throughput compared with a Hugging Face baseline, demonstrating the practical gains of these optimizations for biological foundation model training.
Related
- GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
- BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE
- Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72
- EMO: Frustratingly Easy Progressive Training of Extendable MoE
Source: NVIDIA Developer | 2026-09-24