Multi-Block Diffusion Language Models

arXiv:2606.29215v1 Announce Type: cross Abstract: Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation. A natural next step is to extend them from Single-Block Diffusion (SingleBD) to Multi-Block Diffusion (MultiBD), where a extit{running-set} of consecutive blocks is decoded concurrently for inter-block parallelism. However, existing BD-LMs are mostly trained under teacher forcing, where the model observes only one noisy block conditioned on a clean prefix. While the recent diffusion forcing strategy introduces visibility among multiple noisy blocks, its training states still differ from MultiBD inference, where decoding operates on a bounded extit{running-set} with heterogeneous slot-wise noise patterns. To bridge this gap, we propose extit{Multi-Block Diffusion Language Models} (MBD-LMs), obtained by post-training BD-LMs with extit{Multi-block Teacher Forcing} (MultiTF). MultiTF integrates teacher forcing and diffusion forcing by training on bounded extit{noise-groups} conditioned on clean prefixes, with randomized extit{noise-schedulers} that better match MultiBD inference states. To make MultiBD practically executable, we further introduce an optimized decoding algorithm based on the extit{Block Buffer} mechanism that preserves prefix-cache reuse, keeps input shapes static, and translates increased decoding parallelism into wall-clock acceleration. Empirically, MBD-LLaDA2-Mini increases average Tokens Per Forward pass (TPF) from 3.47 to extbf{6.19} and improves average accuracy from 79.95% to extbf{81.03%}; when combined with DMax, MBD-LLaDA2-Mini-DMax reaches an average TPF of extbf{9.34} with only a 1.02% accuracy drop on math and code benchmarks.

Source: arXiv cs.CL | 2026-06-30