Lost in Phonation: Voice Quality Variation as an Evaluation Dimension for Speech Foundation Models

arXiv:2510.25577v2 Announce Type: replace-cross Abstract: Recent advances in Speech Foundation Models (SFMs) enable direct processing of raw audio, allowing models to respond to subtle paralinguistic variation. However, how these models interpret non-lexical cues remains largely unstudied. We introduce VQ-Bench, a controlled evaluation suite featuring a parallel dataset of synthesized modal, breathy, creaky, and end-creak phonation types. We evaluate SFM sensitivity through open-ended generation across four ecologically valid domains, alongside speech emotion recognition. Our results reveal performance gaps: while a leading commercial API failed basic biometric sanity checks, other models exhibited systematic shifts in agency, empathy, and leadership based on phonation. Our findings also highlight gender asymmetries in salary and leadership endorsements, demonstrating that SFMs may mirror or amplify human social biases. This work establishes a reproducible framework for ensuring responsible paralinguistic interpretation in speech-based AI.

Related

• Voice Mapping of Text-to-Speech Systems: A Metric-Based Approach for Voice Quality Assessment
• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
• What Counts as Real? Speech Restoration and Voice Quality Conversion Pose New Challenges to Deepfake Detection
• SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings

Source: arXiv cs.AI | 2026-08-17