Model Releases
FIRM: Fisher-Informed Retention Merging for Federated LLM Unlearning
arXiv:2610.21720v1 Announce Type: cross Abstract: Federated LLM unlearning addresses targeted knowledge removal when forget and retain data remain distributed across private clients and are inaccessib
arXiv:2610.21720v1 Announce Type: cross Abstract: Federated LLM unlearning addresses targeted knowledge removal when forget and retain data remain distributed across private clients and are inaccessible to the server. The resulting locally trained adapters encode conflicting objectives, making their aggregation fundamentally different from conventional federated optimization. A key failure mode is magnitude explosion, whereby unnormalized cross-client aggregation increases the merged update magnitude and degrades retained utility under stronger unlearning. We propose FIRM (Fisher-Informed Retention Merging), a single-round framework for stable and retention-aware server-side aggregation. FIRM uses client-local Fisher statistics as a unified sensitivity signal to construct a Fisher-normalized base merge and impose explicit control over retain-sensitive parameters. Its second-order retention projection admits a unique minimum-distance solution, while the corresponding dual scales with the number of retain clients rather than the model dimension. Experiments across WMDP, TOFU, and MUSE-HP using Zephyr-7B and Llama-3-8B show utility gains at comparable forgetting on WMDP and MUSE-HP, with metric-dependent trade-offs on TOFU. Controlled comparisons support benefits of the combined base merge and additional retention correction at selected operating points.
Related
- REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference
- FedCVESA: Taking Away Training Data in Federated Learning via Correlation Value Encoding and Segmented Aggregation
- SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoders
Source: arXiv cs.AI | 2026-10-16