Safety
On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs
arXiv:2610.19892v1 Announce Type: cross Abstract: Vision-language models suffer significant accuracy drops under common corruptions. While test-time adaptation (TTA) successfully mitigates these shift
arXiv:2610.19892v1 Announce Type: cross Abstract: Vision-language models suffer significant accuracy drops under common corruptions. While test-time adaptation (TTA) successfully mitigates these shifts in CLIP-like contrastive models, its application to generative multimodal large language models (MLLMs) remains underexplored and uniquely challenging due to their sequential token generation. In this work, we demonstrate that representation-based TTA fails to recover performance in MLLMs. Our experiments reveal that even when violating standard TTA assumptions by leveraging target data offline, sample-blind feature alignment recovers only marginal accuracy. We attribute this failure to the nature of corruption shifts: unlike CLIP-like architectures, which benefit from a shared domain-shift direction within a single pooled classification embedding, MLLM visual token sequence shifts are nearly orthogonal and span the full embedding space.These findings bound what any sample-blind, position-shared correction at a single encoder site can achieve, establishing that representation-space interventions alone might be insufficient to ensure corruption robustness in MLLMs.
Related
- RADC: Risk-Aware Dual Caching for Vision-Language Test-Time Adaptation
- Towards Purified Multi-Label Test-Time Adaptation of Vision-Language Models
- Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selection
- Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation
Source: arXiv cs.CV | 2026-10-15