VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences

arXiv:2609.00921v1 Announce Type: new Abstract: Personalized Large Language Models (PLLMs) aim to tailor responses to individual users, where a central challenge is preference reasoning: inferring query-relevant preferences from user-related history. Existing benchmarks, however, largely assume that such preference can be retrieved from semantically related history. We study an underexplored but practically important regime, profile-preference conceptual misalignment (PRCM), where observable profile cues and query-specific preferences lie in different concept spaces, making semantic retrieval inconsistent for personalization. We introduce VIBE-Bench, a benchmark with two psychology-grounded tasks, 3,504 personas and 12,239 dialogues, including a manually verified gold test set, and requires cross-concept preference reasoning beyond surface semantic overlap. Experiments with several personalization methods show that current PLLMs largely rely on shallow semantic correlations and fail to acquire robust cross-concept mappings. These findings establish PRCM as a distinct failure regime in PLLMs and position VIBE-Bench as a focused testbed for advancing preference reasoning beyond semantic matching.

Related

• Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions
• Efficient Exploration for Iterative Nash Preference Optimization
• Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History
• PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments
• HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs

Source: arXiv cs.AI | 2026-09-02