Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answering

arXiv:2610.16722v1 Announce Type: cross Abstract: Visual contrastive decoding corrects answers using the prediction difference between the original image and a reference condition, yet the same contrast also turns supported abstentions into incorrect answers. We find that abstentions are lost at two stages: contrastive decoding with candidate filtering can exclude the abstention token before any comparison takes place, and the reference branch's response to the abstention instruction is removed when the reference prediction is subtracted. We therefore propose instruction-preserving decoding, which keeps the effect of the abstention instruction on the original-image prediction intact while leaving the contrast strength freely adjustable. Using a perturbed image and an image-free prediction as references, we obtain IP-VCD and IP-M3ID. On decision utility (DU), which credits both correct answers and supported abstentions, both methods outperform all compared non-IP baselines on every one of nine models on VizWiz, and IP-VCD also leads on eight of nine models on Food-101. Candidate measurements and paired per-question comparisons show that computing the instruction effect and the visual contrast term separately lets contrastive decoding support both correct answers and supported abstentions.

Related

• Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition
• Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration
• Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs

Source: arXiv cs.CV | 2026-10-14