Beyond the Name: Demographic Leakage in De-Identified Resumes and Evaluation Artifacts in LLM Bias Audits

arXiv:2609.16501v1 Announce Type: cross Abstract: De-identified resume screening assumes that redacting explicit fields prevents ethnocultural inference; however, recent audits attribute residual leakage to declared languages. We investigate whether eliminating language fields resolves this leakage across nine open-weight models and 620 counterfactual resumes. By holding language attributes strictly identical, we isolate unstructured prose across five ethnocultural conditions and three cue-salience tiers. Target-group recovery averages 0.757 overall and saturates at 1.000 under high salience, demonstrating that non-language prose sustains demographic inference. Crucially, models diverge only under faint cues (0.086-0.690), establishing salience as an essential evaluation axis. Furthermore, pairwise LLM-as-a-judge outcomes are highly sensitive to evaluation design: forbidding ties yields an apparent selection-rate ratio of 0.39 alongside strong position and content effects, whereas permitting ties produces near-universal ties for most models (ge94%). Downstream scoring shows only very small between-condition differences, highlighting the need to distinguish demographic signals recoverable from resume content from effects introduced by the evaluation protocol.

Related

• Fairness Is Not Enough: Auditing Competence and Intersectional Bias in AI-powered Resume Screening
• Bias in the Tails: How Name-conditioned Evaluative Framing in Resume Summaries Destabilizes LLM-based Hiring
• Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval
• PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

Source: arXiv cs.AI | 2026-09-16