Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment

arXiv:2501.19060v4 Announce Type: replace Abstract: Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes. Existing VLM-specific calibration methods mainly rely on textual features of train classes, limiting their applicability across class and distribution shifts. We propose extbf{Shift-Aware Calibration (SAC)}, a training-free, sample-wise method that uses the discrepancy between the output logits of the original and fine-tuned CLIP as a calibration signal. We explicitly call this quantity logit shift, since it is measured in prediction space rather than between hidden representations. SAC maps this logit shift to a positive scaling factor that adjusts confidence while preserving the predicted class. Across 11 datasets and five fine-tuning methods, SAC improves calibration on train and unseen classes and under cross-dataset and domain-generalization evaluation. Mutual Information (MI) analysis shows a statistical dependency between the logit-shift signal and ECE across methods and datasets, providing empirical evidence that the signal used by SAC is informative for calibration and supporting the effectiveness of its shift-aware design.

Related

• UAV-VL-R1: Generalizing Vision-Language Models via Supervised Fine-Tuning and Multi-Stage GRPO for UAV Visual Reasoning
• IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
• Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
• Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning
• Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain

Source: arXiv cs.CV | 2026-07-28