Canaries in the Bank: Auditing User-Level Privacy in Private Evolution

arXiv:2609.13499v1 Announce Type: cross Abstract: Private Evolution (PE) generates high-fidelity synthetic data in federated settings without exposing users' raw data. It aggregates clipped user votes over a shared candidate bank into a differentially private histogram, with noise calibrated to the worst-case user contribution. However, it is unclear whether an adversary can realize this worst-case privacy loss while following the PE protocol. We introduce a protocol-aware empirical audit in which the server commits to a single shared candidate bank and replaces roughly 1% of its entries with probes derived from a known, non-private canary. We evaluate eight attacks, including an unchanged-bank baseline, exact copies, plausible paraphrases, and high-entropy synthetic nonces. Experiments on Yelp and Sentiment140 show that natural-text attacks remain substantially below the theoretical DP bound, while nonce-based attacks yield considerably stronger bounds and come closest to the mechanism's privacy ceiling. These results quantify the gap between formal worst-case privacy and leakage achievable through protocol-valid candidate-bank manipulation.

Related

• Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation
• Amplifying Membership Signal Through Chained Regeneration
• PRIVEE: Privacy-Preserving Vertical Federated Learning Against Feature Inference Attacks
• Have I Seen You? Embedding Behavior Signals Synthetic Face Dataset Membership

Source: arXiv cs.AI | 2026-09-15