Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

arXiv:2610.19407v1 Announce Type: new Abstract: Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.

Which Source Wins? Task-Dependent Reliance in Vision-Language Models
New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models
Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
OmniConfess: Eliciting Token Confessions to Mitigate Omni-Modal Hallucination

Source: arXiv cs.CL | 2026-10-15

Related