Personalized Federated Direct Preference Optimization via LoRA Adapters

arXiv:2610.12885v1 Announce Type: new Abstract: Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice. We propose Personalized FedDPO (pFedDPO), in which each client maintains a small private LoRA adapter atop a shared global base model. In pFedDPO, the base model is synchronized via FedAvg each round, while LoRA adapters remain local and capture client-specific preference patterns. We prove that this decomposition replaces the gradient diversity kappa^2 in the FedDPO bound with a residual heterogeneity kappa^2_{eff}, which is no larger than kappa^2 when adapters absorb client-specific directions, at the cost of an additive adapter-bias term that vanishes as adapters converge. Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per-client reward accuracy.

Related

• FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning
• Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models
• SafeLM: Unified Privacy-Aware Optimization for Trustworthy Federated Large Language Models
• A Unified Framework for Fair and Personalized Decentralized Learning under Communication Constraints

Source: arXiv cs.LG | 2026-10-12