Overfitting Mitigation via Singular Value Decomposition in Minimum Bayes Risk Decoding

arXiv:2609.01135v1 Announce Type: new Abstract: Minimum Bayes Risk (MBR) decoding enables high-quality text generation by selecting the hypothesis that maximizes a utility metric over sampled pseudo-references. However, it is highly susceptible to metric overfitting: it can irregularly inflate the chosen utility metric at the direct expense of other unoptimized evaluation metrics. To mitigate this, we introduce SVD-MBR, which frames the pairwise utility matrix as a noisy information signal. By computing a low-rank approximation via Singular Value Decomposition (SVD) and retaining only the top-k components, we effectively decouple true consensus from metric noise. Experiments demonstrate that SVD-MBR successfully regularizes decoding, yielding substantial gains across a range of generalized metrics. Furthermore, we reveal that this denoising is metric-dependent: neural metrics encode a robust low-rank consensus ideal for SVD, whereas surface-level metrics struggle to separate signal from metric noise.

Related

• Quit While You're Ahead: Quit for Efficient Candidate Generation in Machine Translation Reranking
• Noisy-Channel Minimum Bayes Risk Decoding
• Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding
• Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognition

Source: arXiv cs.CL | 2026-09-02