Research
Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representations
arXiv:2609.03940v1 Announce Type: cross Abstract: Most previous work on speech enhancement (SE) based on masked generative modeling relied on discrete token representations of audio signals, obtained
arXiv:2609.03940v1 Announce Type: cross Abstract: Most previous work on speech enhancement (SE) based on masked generative modeling relied on discrete token representations of audio signals, obtained using neural audio codecs (NACs). However, a recent study has shown that continuous latent representations of NACs can be advantageous for SE in terms of speech quality and intelligibility. In this work, we propose masked autoregressive SE (MARSE), a method for SE based on iterative decoding of masked clean speech frames using continuous NAC representations of speech. In particular, we investigate a set of different decoding policies, ceteris paribus, that is, using the same DNN (a Conformer model), the same NAC (the DAC codec) and the same training setup. The results show that MARSE enables a flexible trade-off between SE performance and computational cost. Audio examples and code are available online.
Related
- UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement
- HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models
- The Equalizer: Introducing Shape-Gain Decomposition in Neural Audio Codecs
Source: arXiv cs.AI | 2026-09-04