On the Price of Privacy for Language Identification and Generation

arXiv:2604.07238v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly trained on sensitive user data, understanding the fundamental cost of privacy in language learning becomes essential. We initiate the study of differentially private (DP) language identification and generation in the agnostic statistical setting, establishing algorithms and matching lower bounds that precisely quantify the cost of privacy. For both tasks, approximate (arepsilon, elta)-DP with constant arepsilon > 0 recovers the non-private error rates: exp(-r(n)) for identification (for any r(n) = o(n)) and exp(-Omega(n)) for generation. Under pure arepsilon-DP, the exponents degrade by a multiplicative factor of min{1, arepsilon}, which we show is tight up to constants. Notably, for generation under pure DP with mild assumptions, the upper bound exp(-min{1,arepsilon} dot Omega(n)) matches the lower bound up to some constants, establishing an optimal rate. Our results show that the cost of privacy in language learning is surprisingly mild: absent entirely under approximate DP, and exactly a min{1,arepsilon} factor in the exponent under pure DP.

Related

• Bi-level Heterogeneous Learning for Time Series Foundation Models: A Federated Learning Approach
• AFL: A Single-Round Analytic Approach for Federated Learning with Pre-trained Models

Source: arXiv cs.LG | 2026-04-10