MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets

arXiv:2308.12067v3 Announce Type: replace-cross Abstract: Multimodal large language models are typically trained in two stages: first pre-training on image-text pairs, and then fine-tuning using supervised vision-language instruction data. Recent studies have shown that large language models can achieve satisfactory results even with a limited amount of high-quality instruction-following data. In this paper, we introduce MM-LIMA, which is fine-tuned on a small dataset comprising only 200 examples, amounting to approximately 6% of the instruction-following data used in the alignment dataset for MiniGPT-4. To achieve this, we first propose several metrics to access the quality of multimodal instruction data. Based on these metrics, we present an effective and trainable data selector to automatically identify and filter low-quality vision-language data. By employing this method, MM-LIMA outperforms the original MiniGPT-4 on various evaluations. Overall, our findings demonstrate that less but high-quality instruction tuning data is efficient in enabling multimodal large language models to generate better output. Our code is available at https://github.com/waltonfuture/InstructionGPT-4.

Related

• Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility
• Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs
• VideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAG
• Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
• On the Spectral Geometry of Cross-Modal Representations: A Functional Map Diagnostic for Multimodal Alignment

Source: arXiv cs.AI | 2026-04-14