FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

arXiv:2610.01620v1 Announce Type: new Abstract: Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create a problem we term subspace fragmentation: local projections interact with data heterogeneity to bias aggregated directions, while aggregation can increase update rank and communication cost. Thus, accurate local gradient compression need not preserve global descent. We propose exttt{FedLore}, which shares a low-rank optimization basis within each round and refreshes it across rounds. The shared basis enables exact aggregation in low-rank coordinates and eliminates the identified projection bias. Subspace refresh allows the accumulated model update to exceed the per-round rank budget. We characterize the aggregation bias and establish an O(T^{-1/2}) stationarity bound for the projected-SGD variant under a global-gradient coverage condition and standard smoothness and variance assumptions, with bounded gradient heterogeneity. Experiments on vision and language tasks, including federated pre-training, show that exttt{FedLore} outperforms the evaluated low-rank adapter baselines and matches or exceeds full-parameter training, while reducing communication and optimizer-state memory.

Related

• RW-LoRA: Communication-Efficient Decentralized LoRA Fine-Tuning via Random Walks
• Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA
• FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Models
• FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints

Source: arXiv cs.AI | 2026-10-02