Generation-Step-Aware Framework for Cross-Modal Representation and Control in Multilingual Speech-Text Models

arXiv:2601.17387v3 Announce Type: replace Abstract: Multilingual speech-text models rely on cross-modal language alignment to transfer knowledge between speech and text, but it remains unclear whether this reflects shared computation for the same language or modality-specific processing. We introduce a generation-step-aware framework for evaluating cross-modal computation that (i) identifies language-selective neurons for each modality at different decoding steps, (ii) decomposes them into language-representation and language-control roles, and (iii) enables cross-modal comparison via overlap measures and causal intervention. Applying our framework to SeamlessM4T, and additionally evaluating its generality on the decoder-only model Qwen2-Audio, we find that cross-modal language-representation alignment is highly model dependent. SeamlessM4T exhibits pronounced generation-step-dependent specialization, where only 5-7% of language-representation neurons are shared across modalities and overlap shifts from same-language to typologically related languages during autoregressive generation, whereas Qwen2-Audio maintains substantially larger cross-modal sharing (45-47%) together with stable, strongly language-specific alignment across decoding steps. In contrast, language-control neurons identified at later decoding steps exhibit progressively stronger cross-modal transfer from speech to text in SeamlessM4T. Together, these results demonstrate that generation-step-aware analysis can reveal both shared and model-specific patterns of cross-modal computation.

Related

• Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Models
• Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment
• Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation

Source: arXiv cs.CL | 2026-08-12