Research
Evaluation of Automatic Speech Recognition Using Generative Large Language Models
arXiv:2604.21928v1 Announce Type: new Abstract: Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based sema
arXiv:2604.21928v1 Announce Type: new Abstract: Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative embeddings, and (3) qualitative classification of errors. On the HATS dataset, the best LLMs achieve 92--94% agreement with human annotators for hypothesis selection, compared to 63% for WER, also outperforming semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder models. Finally, LLMs offer a promising direction for interpretable and semantic ASR evaluation.
Related
- ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation
- How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models
- Spotlights and Blindspots: Evaluation Machine-Generated Text Detection
- Evalet: Evaluating Large Language Models through Functional Fragmentation
- HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models
Source: arXiv cs.CL | 2026-04-24