Hardware
Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton
NVIDIA TensorRT’s multi‑device inference lets a single TensorRT network span multiple GPUs using NCCL‑backed collectives, preserving all standard optimizations and supported from TensorRT 11.0 onward.
NVIDIA TensorRT’s multi‑device inference lets a single TensorRT network span multiple GPUs using NCCL‑backed collectives, preserving all standard optimizations and supported from TensorRT 11.0 onward. The 26.07 release of NVIDIA Dynamo‑Triton incorporates this capability into its backend, allowing one KIND_MODEL instance to own several GPUs and expose a single gRPC endpoint for distributed inference. It was demonstrated on Cosmos 3 Nano video generation—processing 44,160 tokens across eight GPUs—and reduced end‑to‑end latency from ~156 s on one GPU to ~34 s (≈6× speedup) while maintaining the expected quality.
Related
- Scaling AI Inference Across Multiple GPUs Using NVIDIA TensorRT with Multi-Device Inference Support
- When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving
Source: NVIDIA Developer | 2026-09-21