Decoupled DiLoCo for Resilient Distributed Pre-training
DGX agentarXiv:2604.21428v1 Announce Type: new Abstract: Modern large-scale language model pre-training relies heavily on the single program multiple data (SPMD) paradigm, which requires tight coupling across