Hardware
Cut Checkpoint Costs with About 30 Lines of Python and NVIDIA nvCOMP
Training LLMs requires periodic checkpoints — full snapshots of model weights, optimizer states, and gradients — whose storage costs can reach $200,000/month for a 405B model on 128 NVIDIA DGX B200...
Training LLMs requires periodic checkpoints — full snapshots of model weights, optimizer states, and gradients — whose storage costs can reach $200,000/month for a 405B model on 128 NVIDIA DGX B200 GPUs. NVIDIA nvCOMP is a GPU-accelerated lossless compression library that supports both standard algorithms like ZSTD and GPU-native formats like gANS, and integrates directly into Python workflows such as PyTorch or TensorFlow. By compressing checkpoints in-GPU before writing, nvCOMP reduces checkpoint sizes by 21–29% (approximately 1.27× for dense models, 1.40× for MoE), fully hiding compression latency behind storage write operations, with codec choice guided by storage bandwidth — ZSTD for 5–10 GB/s network filesystems and gANS for high-performance storage at 15+ GB/s.
Related
- Running Large-Scale GPU Workloads on Kubernetes with Slurm
- NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining
- Nvidia published DWDP (Distributed Weight-Data Parallelism), a new inference parallelism strategy focused on prefill. It sounds slightly ins…
- RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs
- New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys @lmsysorg and RadixArk @radixark, an…
Source: hardware