Safety
Beyond BFI: The CSI for Enhanced Reliability and Validity in Evaluating LLM Personality Traits
arXiv:2503.20182v2 Announce Type: replace-cross Abstract: As large language models (LLMs) increasingly function as human-like assistants exhibiting human-like personality traits, understanding their b
arXiv:2503.20182v2 Announce Type: replace-cross Abstract: As large language models (LLMs) increasingly function as human-like assistants exhibiting human-like personality traits, understanding their behavioral characteristics becomes essential for responsible AI development. However, existing evaluation efforts, which often adapt human psychological assessments such as the Big Five Inventory (BFI), face two significant limitations. First, these approaches often lack reliability, as minor prompt variations can lead to inconsistent test results. Second, the theoretical foundations of these tools, rooted in human studies, are misaligned with the computational nature of LLMs, thereby limiting their validity in predicting real-world model behavior. To address these limitations, we introduce the Core Sentiment Inventory (CSI), a novel personality trait evaluation instrument designed from the ground up and specifically tailored to the unique characteristics of LLMs. CSI covers both English and Chinese, that implicitly evaluates models' personality traits, providing insightful psychological portraits of LLMs. Extensive experiments demonstrate that: (1) CSI effectively captures nuanced behavioral patterns, revealing significant behavioral variations in LLMs across different languages and contexts; (2) Compared to current evaluation tools, CSI significantly improves reliability, yielding more consistent and robust results; and (3) The correlation between CSI scores and LLMs' real-world outputs exceeds 0.85, demonstrating its strong validity in predicting LLM behavior.
Related
- Vibe Check: Understanding the Effects of LLM-Based Conversational Agents' Personality and Alignment on User Perceptions in Goal-Oriented Tasks
- Mapping how LLMs debate societal issues when shadowing human personality traits, sociodemographics and social media behavior
- Culturally Grounded Personas in Large Language Models: Characterization and Alignment with Socio-Psychological Value Frameworks
- When Can Digital Personas Reliably Approximate Human Survey Findings?
Source: arXiv cs.AI | 2026-08-19