Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation

arXiv:2510.03115v2 Announce Type: replace Abstract: Speech-to-Text Translation (S2TT) systems built from Automatic Speech Recognition (ASR) and Text-to-Text Translation (T2TT) modules face two major limitations: error propagation and the inability to exploit prosodic or other acoustic cues. Chain-of-Thought (CoT) prompting has recently been introduced, with the expectation that jointly accessing speech and transcription will overcome these issues. Analyzing CoT through attribution methods, robustness evaluations with corrupted transcripts, and prosody-awareness, we find that it largely mirrors cascaded behavior, relying mainly on transcripts while barely leveraging speech. Simple training interventions, such as adding Direct S2TT data or noisy transcript injection, enhance robustness and increase speech attribution. These findings challenge the assumed advantages of CoT and highlight the need for architectures that explicitly integrate acoustic information into translation.

Related

• Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems
• Automatic Labelling of Speech Translation Errors
• Entity Binding Failures in Speech LLM Reasoning: Diagnosis and Chain-of-Thought Intervention
• A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition

Source: arXiv cs.CL | 2026-08-20