Federated learning with heavy-tailed gradient noise and communication noise: a variance-reduction based algorithm

arXiv:2606.22466v1 Announce Type: new Abstract: Federated learning (FL) is an emerging distributed machine learning paradigm that enables local devices to jointly train a global model while keeping data decentralized and private. We propose a variance-reduction based algorithm, VRA-FedSGD, for FL in the presence of heavy-tailed gradient noise and communication noise, where these noises are prevalent in large-scale machine learning over wireless networks and Internet of Things deployments. VRA-FedSGD employs a momentum variance reduction technique together with a nonlinear mapping to mitigate heavy-tailed gradient noise, and uses a variance-reduced aggregation mechanism to suppress heavy-tailed communication noise. In the mean sense, VRA-FedSGD achieves a convergence rate of {smallOleft(K^{-(p-1)/(2p-1)}right)} for nonconvex objective functions, where p is the tail index of heavy-tailed noise. In the almost sure sense, VRA-FedSGD achieves a convergence rate of ilde{O}left(K^{-(1-1/(p-epsilon))}right) for strongly convex objective functions, where epsilon is an arbitrarily small constant. Simulated experiments on a logistic regression problem with real-world data verify the effectiveness of VRA-FedSGD.

Source: arXiv cs.LG | 2026-06-23