Robustifying Asynchronous SGD via Soft Throttling

arXiv:2609.39357v1 Announce Type: new Abstract: Asynchronous SGD is a popular algorithm for distributed learning where each client's gradient update is applied on arrival. This leads to a speed-up, but also an increased vulnerability to attacks, as fast clients can dominate the total update. We introduce Throttle, a Byzantine-robust generalization of asynchronous SGD where the key idea is to exponentially down-weight updates from faster clients by a factor q. Both asynchronous SGD (q=1) and synchronous Byzantine-robust SGD (qoinfty) correspond to specific settings of Throttle. We provide a theoretical analysis of the convergence rate and validate the robustness to attacks both theoretically and empirically. Remarkably, our experiments show that this down-weighting mechanism can also improve performance over standard asynchronous SGD even in the non-Byzantine setting.

Related

• Robust stochastic first order methods in heavy-tailed noise via medoid mini-batch gradient sampling
• FedPGT: Progressive Gradient Transmission for Vehicular Federated Learning over Time-Varying Channels
• The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication
• Statistical Inference for Stochastic Gradient Descent Beyond Finite Variance

Source: arXiv cs.LG | 2026-10-01