Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?

arXiv:2610.13550v1 Announce Type: new Abstract: Recent speech LLMs can leverage rich paralinguistic cues, enabling more complex and personalized interactions. However, little work investigates how this sensitivity can lead to stereotyping, and existing evaluations largely measure demographic disparities without distinguishing beneficial personalization from stereotypical behavior. We introduce BASIS-Bench, a benchmark designed to disentangle these two aspects through three axes: (1) demographic robustness, measuring disparities in content-based tasks across speakers and demographic classification capabilities; (2) stereotypical bias, measuring whether perceived demographic cues trigger unprompted stereotypical behavior; and (3) personalization, measuring whether models can tailor responses to demographic characteristics when explicitly requested. We evaluate eight recent speech LLMs and find that, despite often recognizing demographic cues, most fail to translate them into appropriate personalization. Explicit demographic labels improve personalization but also amplify stereotypical associations already present in the textual backbones. These findings highlight the need to train speech LLMs not only to perceive demographic cues, but to learn when they should shape their responses.

Related

• S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
• Beyond Static Personas: Situational Personality Steering for Large Language Models
• PersonaVLM: Long-Term Personalized Multimodal LLMs
• TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

Source: arXiv cs.CL | 2026-10-12