It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at epsilon leq 4/255

arXiv:2609.38298v1 Announce Type: new Abstract: Vision Language Models (VLMs) are widely deployed in safety-critical scenarios, and understanding to which extent they can be controlled by adversarial perturbation is a prerequisite for evaluating their trustworthiness. Existing representation-alignment attacks, which make a VLM perceive a target image, achieve limited success at arepsilon leq 4/255. Therefore, VLMs seems robust to perturbations in this range. We show that this robustness does not hold, as targeted semantic substitution succeeds within the same range. Specifically, we align each stream of the source image with its counterpart in the target image in the victim VLM's post-merger token space, operating under a white-box threat model. We evaluate under a strict success criterion, requiring the model to simultaneously name the target, confirm its presence, and deny the source. In images, target semantics appear at arepsilon = 2/255 and complete replacement reaches 38% at arepsilon = 4/255. On video, complete replacement reaches 35.9% at arepsilon = 1/255. We also observe a phenomenon of extit{semantic fusion}, where Large Language Model (LLM) rationalizes contradictory visual signals into a coherent narrative.

Related

• DIVA: Exploiting Cross-Step Conditional Propagation for Visual Jailbreaks in Discrete Diffusion Vision-Language Models
• Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks
• Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
• SeFaR: Semantic Feature-aware Robustness Testing of Deep Neural Networks

Source: arXiv cs.CV | 2026-10-01