Model Releases
Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passing
arXiv:2610.12711v1 Announce Type: new Abstract: Federated Large Language Models (FedLLMs) are promising to deliver privacy-preserving, collaborative intelligence. The decentralized training process is
arXiv:2610.12711v1 Announce Type: new Abstract: Federated Large Language Models (FedLLMs) are promising to deliver privacy-preserving, collaborative intelligence. The decentralized training process is vulnerable to Byzantine attacks that corrupt semantic consistency and factual resilience through malicious model updates. This paper presents a new resilience strategy that prevents semantically corrupted knowledge from propagating during federated aggregation. A defense problem is formulated to jointly maximize global accuracy and suppress manipulated model updates, enabling trustworthy knowledge aggregation without accessing private local data. To strengthen the resilience of FedLLMs, we develop a novel Hypergraph Message Passing Graph Autoencoder (HMP-GAE) framework, where low-rank model updates are represented as nodes in a hypergraph and high-order semantic relationships among multiple agents are exploited through iterative message passing. By leveraging historical checkpoints and cross-agent relational reasoning, HMP-GAE identifies and suppresses semantically inconsistent updates while preserving knowledge alignment among agents throughout federated training. HMP-GAE is implemented using Qwen-2.5 and Llama-3.2 as pretrained FedLLM backbones and evaluated on AG News and Yahoo! Answers datasets. Compared to existing defenses, HMP-GAE improves global accuracy by 1.21%, reduces final-model classification semantic entropy by 44.08%, and lowers perplexity by 13.34%. HMP-GAE outperforms the GAE-based defense by improving the accuracy 1.39%, reducing the semantic entropy by 54.5%, and lowering perplexity by 35.5%.
Related
- Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs
- FLARE: Adaptive Multi-Dimensional Reputation for Robust Client Reliability in Federated Learning
- CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
Source: arXiv cs.LG | 2026-10-12