Approximation in Contrastive Representation Learning

arXiv:2610.15388v1 Announce Type: new Abstract: Contrastive representation learning (CRL) is a fundamental paradigm for learning transferable representations. Recent work identifies its population target as the pointwise mutual information (PMI) logfrac{p(x,y)}{p_X(x)p_Y(y)} between two modalities X,Y, where p is the joint density function and p_X,p_Y are the marginals. However, it remains unclear why standard inner-product scores f(x)^op g(y) can effectively approximate a generally nonseparable population target PMI. In this paper, we address this question from the aspect of approximation for CRL. By linking PMI to a compact operator, we show that D-dimensional inner-product models perform rank-D spectral approximation, with error controlled by the spectral tail. We then study neural network realizations: in general settings, we show that deep neural network scoring functions are universally consistent, and the approximation error rates depend on the smoothness of the PMI function; under structured low-rank data models we obtain improved rates attained by moderate-width encoders. Finally, we establish a fast calibration rate from contrastive excess risk to downstream retrieval error under a density-ratio Tsybakov noise condition.

Related

• Statistical Consistency and Generalization of Contrastive Representation Learning
• Patient-Aware Contrastive Learning Preserves Per-Patient Structure in RR-Interval Representations
• Consistency of augmentation graph and network approximability in contrastive learning

Source: arXiv cs.LG | 2026-10-14