Reliability Challenges in Diffusion Vision-Language Models

arXiv:2609.01318v1 Announce Type: cross Abstract: Diffusion-based Large Vision-Language Models (dLVLMs) have recently emerged as a compelling alternative to autoregressive (AR) LVLMs, offering advantages in parallel decoding, bidirectional context, and controllable generation. Despite rapid progress, their reliability properties remain largely uncharacterized. We present the first systematic reliability evaluation of hallucination and bias in dLVLMs, benchmarking six diffusion models against competitive AR baselines across four dimensions. Our key findings are: (1) dLVLMs reverse the yes-bias of AR models in binary visual queries; (2) they achieve competitive hallucination rates yet exhibit degraded linguistic quality; (3) they collapse to near-zero accuracy on underrepresented racial groups with opposite-polarity gender bias; and (4) they exhibit accuracy collapse in multiple-choice settings when the correct option is shorter than its distractors, associated with a length prior that emerges at the first denoising step. Tokens committed at late denoising steps with low confidence further correlate with hallucinated content, pointing to a mechanistic signal unique to diffusion generation. These patterns vary across model families, suggesting reliability is shaped by the generative paradigm together with training data.

Related

• Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models
• Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions
• MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Models
• S-GRPO: Unified Post-Training for Large Vision-Language Models

Source: arXiv cs.CL | 2026-09-02