Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMs

arXiv:2610.18141v1 Announce Type: new Abstract: Omnimodal large language models (OLLMs) are susceptible to cross-modal hallucinations under audio-visual interference. We find that, for a substantial fraction of incorrect joint predictions, the question-relevant unimodal branch can still recover the correct answer, suggesting that task-relevant evidence can remain recoverable from the relevant modality even when it is not reflected in the joint prediction. We refer to this recoverable failure pattern as modality misrouting. We propose CaRe (Conflict-aware Residual Intervention), a training-free inference-time method for identifying and correcting modality misrouting. CaRe uses a text-only modality probe to identify a candidate modality-specific counterfactual branch, and flags potential misrouting when its prediction conflicts with the original joint prediction. For such cases, CaRe computes a residual direction from the discrepancy between the counterfactual and fused representations and uses it to steer the original multimodal state toward the relevant modality evidence, while retaining both modalities in the original audio-visual inference path. The intervention strength is adapted based on the confidence of the counterfactual branch. Experiments on CMM and AVHBench across two OLLMs show that CaRe consistently outperforms the evaluated inference-time hallucination-mitigation baselines.

Related

• Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation
• Probing Cross-modal Information Hubs in Audio-Visual LLMs
• Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
• Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory
• The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs

Source: arXiv cs.AI | 2026-10-15