Real-Time Performance Monitoring and Faster Debugging with NCCL Inspector and Prometheus

NCCL Inspector is a profiling plugin that provides detailed, per-communicator, per-collective performance and metadata logging, designed to help users analyze and debug NCCL collective operations by generating structured JSON output . The tool integrates with Prometheus through a Prometheus format exporter output and Grafana templates for dashboard rendering . The Inspector operates as a lightweight profiling tool that intercepts and analyzes NCCL calls without modifying application code, capturing metrics including bandwidth utilization, message sizes, and communication patterns .

Source: NVIDIA Developer | 2026-05-07