Representation Alignment from a Few Paired Data via Distilled Optimal Transport

arXiv:2610.12781v1 Announce Type: new Abstract: Aligning two independently pretrained unimodal encoders typically demands millions of paired examples. In this paper, we study the regime where only N sim 100--500 pairs are available, while unpaired data on either side is abundant. Our method expresses every representation by its similarities to the N paired anchors, which places both modalities in a common coordinate system; matches the two point clouds with a single entropic optimal transport problem solved over the entire pool, whose marginal constraints counteract the hubness that defeats nearest-neighbor matching; and distills the resulting plan into two lightweight encoders, so that the correspondence becomes inductive. We additionally put this marginal constraint in relation with the one-step hub corrections of the bilingual lexicon induction literature. Pairing is required only for the anchors: the unpaired collections may be drawn independently from the two marginals, so that no item outside the anchor set needs to have both of its modalities observed. We evaluate on image-text and molecule-text alignment, where our method outperforms existing approaches at the same anchor budget on Flickr30k, MS-COCO and ChEBI-20.

Related

• Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport
• SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport
• Learning Relative Representations for Fine-Grained Multimodal Alignment with Limited Data
• Representational Alignment Across Model Layers and Brain Regions with Multi-Level Optimal Transport

Source: arXiv cs.AI | 2026-10-12