Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models

arXiv:2608.27531v1 Announce Type: cross Abstract: The safety of large vision-language models is increasingly stress-tested by multimodal jailbreaks, yet existing attacks remain largely static at the meta level: template-based attacks freeze the image--text layout, while iterative attacks adapt only the image--text content with fixed attack strategies and frozen attacker parameters. We propose Meta-Adaptive Multimodal Jailbreaking (MAMJ), which instead optimizes the attacker itself along two axes: an attack strategy prompt (ASP) heta governing attack iteration and attacker weights phi determining attack effectiveness. Across groups of multimodal attack trajectories, an LLM-based critique first refines heta, after which group-aggregated attack-success-rate (ASR) rewards update phi. On MM-SafetyBench, MAMJ achieves 81.0%, 78.9%, and 82.3% ASR against GPT-4o, Gemini-3-Pro-Preview, and Seed 2.0, respectively, outperforming the strongest sample-level baseline by up to 24.1 percentage points. The learned attacker (heta^star,phi^star) also transfers without retraining to unseen victims and remains effective under representative defenses. These results reveal a systemic vulnerability of frontier VLMs to meta-adaptive jailbreaks and motivate defenses against meta-level adversaries. Code is available at https://github.com/Alibaba-VELLDEPTH/MetaJailbreak-VLM.

Related

• A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation
• Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderation
• Jailbreaking Frontier Foundation Models Through Intention Deception
• Babel: Jailbreaking Safety Attention via Obfuscation Distribution Optimized Sampling

Source: arXiv cs.CV | 2026-08-31