Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment

arXiv:2610.22049v1 Announce Type: cross Abstract: Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information. Naturalistic speech offers a rich testbed for this challenge: it unfolds over multiple timescales, but fMRI's noisy, temporally smeared measurements make this structure hard to recover. We introduce NOEMA, a temporal convolutional decoder that aligns windows of consecutive fMRI measurements with multimodal stimulus embeddings. These embeddings combine complementary text, audio, and speech representations of the same stimulus segment, enabling its retrieval directly from brain activity. On two naturalistic speech datasets, Cephalonauts One and LeBel, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of held-out queries. This substantially outperforms baseline decoders, an advantage that persists when the candidate pool is extended with thousands of distractors absent from training. Controlled ablations establish the contributions of temporal modeling and complementary stimulus representations.

Related

• A cross-species neural foundation model for end-to-end speech decoding
• Decomposing Brain-LLM Alignment with Interpretable Semantic Representations
• Cross-Subject Semantic Decoding with Shared-Space Alignment for Generalized Neural Representation Learning

Source: arXiv cs.AI | 2026-10-16