When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Models

arXiv:2604.23994v1 Announce Type: cross Abstract: Discrete diffusion language models (dLLMs) enable parallel token updates with bidirectional attention, yet practical generation typically adopts blockwise semi-autoregressive decoding. This switch creates a training-inference mismatch: training denoises with full-sequence context, while inference commits tokens within a bounded block without future context. Therefore, decoding with fixed-size or heuristic-based blocks can lead to premature token commitments, as decisions are made without full access to future context that could alter those choices. Motivated by this, we propose self-containedness as a principled criterion for block commitment. A block is self-contained if its predictions remain consistent with Future-Aware (FA) or without No-Future (NF) access to future context, reframing block boundary selection as a test of self-containedness rather than a heuristic choice. Based on this principle, we introduce Variable-size Self-contained Blocks (VSB) for dLLMs. VSB scores and selects block boundaries using the divergence between token-level predictive distributions under NF and FA conditioning, which quantifies how predictions would change if future context were revealed. We provide theoretical justification linking self-containedness to predictive consistency, and extensive experiments validate VSB's efficacy over fixed-size and heuristic blockwise decoding.

Related

• Stability-Weighted Decoding for Diffusion Language Models
• Diffusion Language Models Know the Answer Before Decoding
• LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Models
• Remask, Don't Replace: Token-to-Mask Refinement in Masked Diffusion Language Models
• Accelerating Speculative Decoding with Block Diffusion Draft Trees
• Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models

Source: arXiv cs.CL | 2026-04-28