FedMIX-P: Mixing Local and Global Preconditioners for Federated Vision and Language Model Training

arXiv:2610.01515v1 Announce Type: cross Abstract: Adaptive preconditioners accelerate model training, but heterogeneous client geometries can bias federated updates even when gradients are evaluated at the same model. Round-start synchronization alone cannot prevent this mismatch from reappearing during local training. We propose exttt{FedMIX-P}, which mixes shared and local preconditioners at every local step, retaining local adaptation while reducing mean-squared operator mismatch by a factor of lambda^2. For smooth nonconvex objectives with stochastic gradients and partial participation, we establish an O(R^{-1/2}) stationarity bound using suitable stepsizes and a horizon-dependent mixing weight, without requiring local preconditioners to converge to one another. A two-client counterexample shows that fixed positive mixing can preserve a nonstationary fixed point. The theory covers bounded linear symmetric positive-definite preconditioners. Experiments with SOAP, Sophia, and Muon variants across vision and language tasks show improvements over corresponding local optimizers, including accuracy gains of up to 19.47 percentage points and lower validation loss for 60M--350M language models. Full nonlinear and momentum-based updates require separate analysis.

Related

• When Is Shallow Enough? Adaptive Split Federated Learning with Client-Specific Sufficiency Estimation
• Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
• How Does Local Landscape Geometry Evolve in Language Model Pre-Training?
• Why Muon Outperforms Adam: A Curvature Perspective

Source: arXiv cs.AI | 2026-10-02