Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach

arXiv:2605.20674v1 Announce Type: new Abstract: We introduce CoMET, extit{extbf{C}omposing extbf{M}odality extbf{E}ncoders with extbf{T}abular foundation models}, a simple yet highly competitive method for multimodal classification: pass each modality through a frozen pre-trained backbone, compress the resulting embeddings with PCA, and concatenate as input into a Tabular Foundation Model (TFM) for prediction. We show that PCA alone suffices to act as an adaptor yielding strong, robust performance across modalities. When the exttt{CLS} tokens of the foundation model align poorly with downstream tasks, we propose extbf{PALPooling}, a lightweight adaptive token pooler that consistently improves representation quality. By composing strong frozen representation learning backbones with TFMs, our approach achieves state-of-the-art results across diverse multimodal benchmarks without any training. On hierarchical tasks with large fine-grained class spaces, our approach enables fast and scalable classification, handling datasets with over 500,000 samples and 2,000 classes without any fine-tuning. Overall, our results show that the composition of foundation models is a simple, yet powerful, out-of-the-box solution for multimodal learning, challenging the necessity of complex, end-to-end training pipelines for new problems.

Source: arXiv cs.LG | 2026-05-21