Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weighting

arXiv:2510.02913v2 Announce Type: replace Abstract: Vision-language models such as CLIP demonstrate impressive zero-shot generalization but remain highly vulnerable to adversarial attacks. Prior adversarial methods treat all samples equally in the loss function, despite the fact that not all inputs contribute equally to adversarial vulnerability. Some samples retain high confidence even under large perturbations, while others are unstable and change predictions with minor noise. To address this, we propose Confidence-Aware Weighting (CAW) to enhance the zero-shot robustness of vision-language models. CAW introduces two key components: (1) a Confidence-Aware Loss that prioritizes uncertain adversarial examples by scaling the KL divergence between clean and adversarial predictions, and (2) a Feature Alignment Regularization that preserves semantic consistency by minimizing the distance between frozen and fine-tuned image encoder features on adversarial inputs. Together, these components improve both clean and robust accuracy without compromising generalization. Extensive experiments on TinyImageNet and 14 additional datasets show that CAW outperforms recent state-of-the-art methods such as PMG-AFT and TGA-ZSR under strong adversarial attacks like AutoAttack, while also being more memory-efficient.

Related

• Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP
• Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models
• TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models
• Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model
• Calibrating Uncertainty for Zero-Shot Adversarial CLIP

Source: arXiv cs.CV | 2026-07-28