Safety
No PUN Intended: Plausible Unknown Names for Person-Centred LLM Evaluation
arXiv:2608.21206v1 Announce Type: cross Abstract: Person names are widely used as prompt variables in LLM evaluations of factuality, privacy leakage, bias and abstention, but when a name's evidential
arXiv:2608.21206v1 Announce Type: cross Abstract: Person names are widely used as prompt variables in LLM evaluations of factuality, privacy leakage, bias and abstention, but when a name's evidential status is uncontrolled, measurements may conflate memorisation, retrieval, name priors and wrong-person attribution. We operationalise an unknown name as one with plausible First-Last form, no indexed full-name evidence, and no ambiguity signals under a documented validation run, and introduce PUN (Plausible Unknown Names), a protocol for constructing and validating such names, combining Wikidata-derived components, web-enabled LLM screening, and controlled search revalidation. We report acceptance rate, reproducibility, ablations, and a 204-participant human study, finding accepted names are more name-like than controls while participants recover person evidence in only 3% of cases. We release 300 names with comparison controls.
Related
- An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments
- Status Association Does Not Reliably Predict Decision Leakage
- Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs
Source: arXiv cs.AI | 2026-08-24