Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models

arXiv:2602.02043v2 Announce Type: replace-cross Abstract: Modern vision-language models struggle with basic compositional reasoning, failing to bind attributes to objects or relations to their referents. Existing benchmarks either rely on noisy real images that conflate confounding visual variables with the reasoning failure, or use simplistic synthetic scenes lacking the realism modern VLMs are tuned for. We introduce extbf{Auto-Comp}, a fully automated, concept-driven pipeline that bridges this gap by generating photorealistic compositional benchmarks at scale. Its core innovation is a extit{parallel A/B construction}: for each concept, the pipeline emits a extit{Minimal} sample (template caption, isolated objects on a white background) and a extit{Contextual} sample (LLM-rewritten caption, objects embedded in a realistic scene), isolating core binding ability from visio-linguistic complexity. We instantiate extit{four} task families spanning the two canonical axes of compositional binding: extit{Color} and extit{Shape-Color} (attribute binding), and extit{Position} and extit{Relative Size} (relational binding). We evaluate over 25 VLMs spanning CLIP, SigLIP, hard-negative-trained, and frontier generative models. The findings are consistent across architectures and scales: every model exhibits a large Swap-vs-Confusion gap, with low-entropy distractors (e.g., repeated objects or colors) exposing failures extit{beyond} the known bag-of-words limitations. We further uncover a task-dependent trade-off: visio-linguistic context aids relational reasoning but hinders attribute binding through visual clutter. We publicly release the pipeline and benchmarks.

Related

• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
• Binding Visual Features Point by Point
• Understanding the Effects of Distractors on Reasoning Vision-Language Models

Source: arXiv cs.AI | 2026-09-28