CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising

arXiv:2607.28236v1 Announce Type: cross Abstract: Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embedding by jointly optimizing contrastive and reconstruction objective to learn perturbation-invariant representation. We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE

Related

• Developmental approach reveals the statistical learning of Neural Language Models: Transformers generalize from the most abstract statistical patterns
• Learning is Forgetting: LLM Training As Lossy Compression
• An Information-theoretic Propagation Denoising and Fusion Framework for Fake News Detection

Source: arXiv cs.CL | 2026-07-31