Beyond EER: Multi-Dimensional Evaluation of Information Leakage in Speaker De-Identification

arXiv:2609.18673v1 Announce Type: cross Abstract: Speaker de-identification (SDID) aims to preserve privacy by concealing speaker identity while maintaining speech utility. However, current evaluations often reduce privacy to a single dimension - biometric verification performance - typically measured by Equal Error Rate (EER). This narrow focus ignores critical leakage channels, such as soft biometric inference, embedding-level re-identification, and structural template similarity, which threaten the unlinkability and irreversibility of biometric references. We propose a holistic evaluation framework across five complementary metrics: (i) EER, (ii) soft biometric leakage score , (iii) cumulative match characteristic re-identification analysis, (iv) canonical correlation analysis and Procrustes embedding alignment, and (v) intelligibility via word error rate and semantic similarity. Evaluating five SDID systems from the IARPA ARTS program, we demonstrate that these metrics capture independent dimensions of information leakage. Our results indicate that reliance on a single metric can misrepresent the privacy properties of an SDID system.

Related

• TokAN: Accent Normalization Using Self-Supervised Speech Tokens
• Beyond the Name: Demographic Leakage in De-Identified Resumes and Evaluation Artifacts in LLM Bias Audits
• No PUN Intended: Plausible Unknown Names for Person-Centred LLM Evaluation
• Time-Frequency Consistency Learning for Robust Speech Deepfake Detection

Source: arXiv cs.AI | 2026-09-17