The threat of analytic flexibility in using large language models to simulate human data

arXiv:2509.13397v3 Announce Type: replace-cross Abstract: Social scientists are now using large language models to create "silicon samples": synthetic datasets intended to stand in for human respondents. However, producing these samples requires many analytic choices, including model selection, sampling parameters, prompt format, and the amount of demographic or contextual information provided. Across two studies, I examine whether these choices materially affect correspondence between silicon samples and human data. In Study 1, I generated 252 silicon-sample configurations for a controlled case study using two social-psychological scales, evaluating whether configurations recovered participant rankings, response distributions, and between-scale correlations. Configurations varied substantially across all three criteria, and configurations that performed well on one dimension often performed poorly on another. In Study 2, I extended this analysis to a published silicon-sample use case by re-examining Argyle et al.'s (2023) Study 3 using 66 alternative configurations. Correlations between human and silicon association structures differed substantially across configurations, from r = .23 to r = .84. Taken together, the results from these studies demonstrate that different defensible configuration choices can materially alter conclusions about the fidelity of silicon samples. I call for greater attention to the threat of analytic flexibility in using silicon samples and outline strategies that researchers may adopt to reduce this threat.

Related

• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
• Restoring Heterogeneity in LLM-based Social Simulation: An Audience Segmentation Approach
• Distribution Shift Alignment Helps LLMs Simulate Survey Response Distributions
• Evaluating LLMs as Human Surrogates in Controlled Experiments
• Overstating Attitudes, Ignoring Networks: LLM Biases in Simulating Misinformation Susceptibility

Source: arXiv cs.AI | 2026-04-20