Distilling Vision Transformers for Distortion-Robust Representation Learning

arXiv:2604.22529v1 Announce Type: new Abstract: Self-supervised learning has achieved remarkable success in learning visual representations from clean data, yet remains challenging when clean observations are sparse or not available at all. In this paper, we demonstrate that pretrained vision models can be leveraged to learn distortion-robust representations, which can then be effectively applied to downstream tasks operating on distorted observations. In particular, we propose an asymmetric knowledge distillation framework in which both teacher and student are initialized from the same pretrained Vision Transformer but receive different views of each image: the teacher processes clean images, while the student sees their distorted versions. We introduce multi-level distillation that aligns global embeddings, patch-level features, and attention maps and show that the student is able to approximate clean-image representations despite never directly accessing clean data. We evaluate our approach on image classification tasks across several datasets and under various distortions, consistently outperforming existing alternatives for the same amount of human supervision.

Related

• Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR
• ActDistill: General Action-Guided Self-Derived Distillation for Efficient Vision-Language-Action Models
• jBOT: Semantic Jet Representation Clustering Emerges from Self-Distillation
• PASTA: Vision Transformer Patch Aggregation for Weakly Supervised Target and Anomaly Segmentation

Source: arXiv cs.CV | 2026-04-27