Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization

arXiv:2604.22885v1 Announce Type: cross Abstract: Federated cross-modal retrieval faces severe challenges from heterogeneous client data, particularly non-IID semantic distributions and missing modalities. Under such heterogeneity, a single global model is often insufficient to capture both shared cross-modal knowledge and client-specific characteristics. We propose RCSR, a personalization-friendly federated framework that integrates prototype anchoring, retrieval-centric semantic routing, and optional client-specific adapters. Built on a frozen CLIP backbone, RCSR leverages lightweight shared adapters for global knowledge transfer while supporting efficient local personalization. Prototype anchoring helps unimodal clients align with global cross-modal semantics, and a server-side semantic router adaptively assigns aggregation weights based on retrieval consistency to mitigate alignment drift during heterogeneous updates. Extensive experiments on MS-COCO, Flickr30K, and other benchmarks show that RCSR consistently improves global retrieval accuracy and training stability, while further enhancing client-level retrieval performance, especially for clients with incomplete modalities. Code is available at https://github.com/RezinChow/RCSR-Retrieval-Centric-Semantic-Routing.

Related

• Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection
• Handling Missing Modalities in Multimodal Survival Prediction for Non-Small Cell Lung Cancer
• Data-Free Contribution Estimation in Federated Learning using Gradient von Neumann Entropy
• EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models
• Sherpa.ai Privacy-Preserving Multi-Party Entity Alignment without Intersection Disclosure for Noisy Identifiers
• A Taxonomy and Resolution Strategy for Client-Level Disagreements in Federated Learning

Source: arXiv cs.AI | 2026-04-28