Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling

arXiv:2604.13054v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved rapid progress, yet their scaling behavior remains less clearly characterized and often less predictable than that of text-only LLMs. Increasing model size and task diversity often yields diminishing returns. In this work, we argue that the primary bottleneck in multimodal scaling is not task format, but knowledge density in training data. We first show that task-specific supervision such as Visual Question Answering (VQA) contributes little incremental semantic information beyond image captions: VQA signals can be reconstructed from captions with negligible performance loss. We then demonstrate that increasing knowledge density -- through structured caption enrichment and cross-modal knowledge injection -- leads to consistent performance improvements across multimodal and downstream benchmarks. Across controlled experiments, performance correlates more strongly with semantic coverage than with task diversity. These findings suggest that current MLLMs fail to scale primarily because training data lacks sufficient knowledge coverage. We advocate for knowledge-centric multimodal training as a principled foundation for scalable multimodal models.

Related

• VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
• Indexing Multimodal Language Models for Large-scale Image Retrieval
• Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models
• Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies

Source: arXiv cs.CL | 2026-04-16