NoLoCo: No-all-reduce Low Communication Training Method for Large Models
arXiv:2506.10911v2 Announce Type: replace Abstract: Training large language models is generally done on clusters containing thousands of accelerators, communicating over a high-bandwidth interconnect.