Hardware
Another great paper from NVIDIA. They find that only 0.6% to 1.2% of model weights change after each RL step in the six models they measured…
Another great paper from NVIDIA. They find that only 0.6% to 1.2% of model weights change after each RL step in the six models they measured. Yet a standard refit copies the full checkpoint to the rol
Another great paper from NVIDIA. They find that only 0.6% to 1.2% of model weights change after each RL step in the six models they measured. Yet a standard refit copies the full checkpoint to the rollout cluster after every update. For a 1T model across two AWS regions, that copy takes 87.5 minutes. NVIDIA's NeMo-DCR sends only the changed values and still gives the rollout cluster the exact same bits as a full copy. It maps changes from training shards straight into the checkpoint layout, encodes them as XOR masks or overwrites, and streams them through a relay tree while the delta is still being built. A joint commit and retries handle failures in the middle of a transfer. A 1T refit at a 3% change rate takes 150 seconds instead of 87.5 minutes. Across 30B to 1T models, it is 12x to 40x faster than full-checkpoint transfer. Paper: https://academy.dair.ai/papers/nemo-dcr-bit-exact-delta-compressed-refit-for-scalable-agentic-rl-at-trillion-pa-2610.08430
Related
Source: DAIR.AI (X) | 2026-10-08