Model Releases
Evaluating VQA in Vision Language Models using Cooperative Principles
arXiv:2610.02878v1 Announce Type: new Abstract: We evaluate the performance of Vision Language Models in Visual Question Answering (VQA) when questions violate Grice's maxims. To do this, we use VLMs
arXiv:2610.02878v1 Announce Type: new Abstract: We evaluate the performance of Vision Language Models in Visual Question Answering (VQA) when questions violate Grice's maxims. To do this, we use VLMs to generate question modifiers that add non-essential, ambiguous or false information and show that in the presence of such violations, the VLMs that we evaluate (ChatGPT, Claude, Gemini and Llava) show diminished performance. Further, we empirically show the difference between how humans reason pragmatically compared to VLMs, and the difference in VLM reasoning when it resolves violations that are human-induced compared to those that are AI-generated. Finally, we show that human cognitive effort (measured through time-on-task in an experiment) is lower for resolving VLM-induced violations, but VLMs themselves perform less accurately in such cases.
Related
- Who is the richest club in the championship? Detecting and Rewriting Underspecified Questions Improve QA Performance
- Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting
- WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
Source: arXiv cs.CL | 2026-10-05