Stability-Weighted Decoding for Diffusion Language Models

arXiv:2604.17068v1 Announce Type: new Abstract: Diffusion large language models (dLLMs) enable parallel text generation by iteratively denoising a fully masked sequence, unmasking a subset of masked tokens at each step. Existing decoding strategies rely on static confidence metrics computed at a single denoising step, ignoring temporal history and often leading to premature unmasking of unstable tokens. In this work, we theoretically establish that a token's temporal instability, quantified by the KL divergence between consecutive prediction distributions, provides a strict lower bound on its mutual information with the remaining masked context, indicating that temporally unstable tokens are inherently unsafe to unmask. Based on this insight, we propose Stability-Weighted Decoding (SWD), a training-free, plug-and-play strategy that incorporates temporal stability into token scoring and acts as a universal modulator for arbitrary score-based decoding policies. Experiments on code generation and mathematical reasoning benchmarks demonstrate that SWD consistently improves generation accuracy across representative scoring metrics and selection policies, and exhibits exceptional robustness, maintaining a significant performance lead over standard baselines across varying acceleration ratios.

Related

• LogicDiff: Logic-Guided Denoising Improves Zero-Shot Reasoning in Masked Diffusion Language Models
• Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
• Lost in Diffusion: Uncovering Hallucination Patterns and Failure Modes in Diffusion Large Language Models
• Attention-Based Sampler for Diffusion Language Models
• LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Models
• Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow

Source: arXiv cs.CL | 2026-04-21