Research
QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing
arXiv:2608.13119v1 Announce Type: new Abstract: Typographic attacks pose a critical threat to vision-language models (VLMs) by injecting misleading text into images and causing models to rely on adver
arXiv:2608.13119v1 Announce Type: new Abstract: Typographic attacks pose a critical threat to vision-language models (VLMs) by injecting misleading text into images and causing models to rely on adversarial textual cues rather than visual evidence. Existing defenses often require model-specific modifications, additional training, or access to internal model components, limiting their applicability to modern closed-source VLMs. In this paper, we propose QuISE, a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing. QuISE first identifies text regions likely to affect the current query through influence-aware text localization. QuISE then replaces these regions with two semantically distinct replacement texts that are irrelevant to both the query and the image. The final answer is determined by answer consistency across the edited images. Extensive experiments on three typographic-attack benchmarks, four attack settings, and four VLMs show that QuISE consistently improves defended accuracy. QuISE achieves a recovery rate of 67.9-75.0% with a harm rate of 0.5-1.1%.
Related
- GeoThreat: Transferable Targeted Adversarial Attacks on Large Vision-Language Models for Remote Sensing Image Interpretation
- Two Sides of the Same Coin: Co-Evolving Search for Cross-Task Attacks on Vision-Language Models
- High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models
Source: arXiv cs.CV | 2026-08-14