Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism

Nonuniform Tensor Parallelism (NTP) enables large-scale LLM training jobs to dynamically adapt tensor parallelism degree in response to transient GPU unavailability, ensuring sustained Goodput and minimizing computational stalls and throughput loss. NTP incorporates efficient, overlapped tensor resharding techniques that limit overhead to less than 1%, preserving near-optimal compute efficiency and enabling resilient operation on NVIDIA Blackwell systems with up to 72 GPUs. The approach handles device unavailability and resource fluctuations in tightly coupled clusters by dynamically adjusting tensor partition strategies rather than stalling entire jobs.

Source: NVIDIA Developer | 2026-07-06