Rethinking Cross-Modal Fine-Tuning: Optimizing the Interaction Between Feature Alignment and Target Fitting

arXiv:2601.18231v4 Announce Type: replace Abstract: Adapting pre-trained models to unseen feature modalities has become increasingly important due to the growing need for cross-disciplinary knowledge integration. A key challenge here is how to align the representation of new modalities with the most relevant parts of the pre-trained model's representation space to enable accurate knowledge transfer. This requires combining feature alignment with target fine-tuning, but uncalibrated combinations can exacerbate misalignment between the source and target feature-label structures and reduce target generalization. Existing work, however, lacks a theoretical understanding of this critical interaction between feature alignment and target fitting. To bridge this gap, we develop a principled framework that establishes a provable generalization bound on the target error, which explains the interaction between feature alignment and target fitting through a novel concept of feature-label distortion. This bound offers actionable insights into how this interaction should be optimized for practical algorithm design. The resulting approach achieves significantly improved performance over state-of-the-art methods across a wide range of benchmark datasets.

Related

• Cross-Modal Bayesian Low-Rank Adaptation for Uncertainty-Aware Multimodal Learning
• Fine-tuning Factor Augmented Neural Lasso for Heterogeneous Environments
• Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
• PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection
• TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models

Source: arXiv cs.LG | 2026-04-21