Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs

arXiv:2609.11762v1 Announce Type: new Abstract: Per-layer differential privacy (DP) clipping improves gradient fidelity in federated learning by allocating per-matrix clipping budgets proportional to parameter count. We show that this recipe breaks for speech large language models (speech-LLMs), when the acoustic encoder and the language decoder differ by an order of magnitude in update norm. Single-pool per-layer methods suffer cross-component budget collapse, dragging word error rate (WER) far from flat global clipping or collapsing training entirely. When the norm imbalance is milder, adaptive single-pool methods partially recover, confirming that collapse severity scales with the inter-component norm ratio. We empirically diagnose the root cause across six per-layer methods and three speech-LLM architectures. We then propose alpha-split, a two-pool allocation that normalises encoder and LLM parameters into independent pools, and show that joint ell_2 sensitivity and the original (arepsilon,elta)-DP guarantee are unchanged. At architecture-calibrated alpha, our method recovers WER utility compared to flat DP, while granting the encoder 4.47{imes} tighter per-component noise protection against speaker voice-based gradient-inversion attacks at only +2.6% LLM noise overhead.

Related

• SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies
• TriShield: Zero-Utility-Loss Defense Against Privacy Backdoors in Federated Language Model Fine-Tuning via Orthogonal Gradient Projection and Optimizer State Entanglement
• ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoning

Source: arXiv cs.CL | 2026-09-11