Research
Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models
arXiv:2509.22415v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have achieved strong vision-language performance, yet their token-level visual evidence remains diffi
arXiv:2509.22415v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have achieved strong vision-language performance, yet their token-level visual evidence remains difficult to inspect. Recent logit-lens attribution methods project each visual-token hidden state into the vocabulary space to explain generated words, but this token-wise readout introduces a mismatch: visual tokens are context-mixed by the model, while the attribution score is decoded independently at each token location. This often produces fragmented attribution maps and can be further affected by autoregressive context signals from preceding text tokens. We propose ERCR, an attribution framework built from Evidence Recomposition (ER) and Predictive Context Residualization (PCR). ER aggregates target evidence across multiple views with different token-to-region assignments, reducing attribution fragmentation caused by a single readout grid. PCR estimates a preceding-token context map with RBO-based rank relevance and subtracts its fitted component from the ER map to suppress context-token interference. Experiments on LLaVA, Qwen2-VL, and InternVL families across COCO Caption, GranDf, and OpenPSG show that ERCR improves visual evidence for target tokens and mitigates preceding-token context interference under the existing evaluation protocol. On Qwen2-VL-2B, ERCR improves TAM F1-IoU from 39.10 to 44.45 on COCO Caption and from 30.83 to 37.20 on GranDf. Overall, ERCR provides a practical refinement for token-level visual evidence inspection.
Related
- Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attribution
- Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models
- SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering
- TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning
Source: arXiv cs.AI | 2026-07-15