Understanding Task Transfer in Vision-Language Models

arXiv:2511.18787v2 Announce Type: replace Abstract: Vision-Language Models (VLMs) perform well on multimodal benchmarks but lag behind humans and specialized models on visual perception tasks like depth estimation or object counting. Finetuning on one task can unpredictably affect performance on others, making task-specific finetuning challenging. In this paper, we address this challenge through a systematic study of task transferability. We examine how finetuning a VLM on one perception task affects its zero-shot performance on others. We introduce Perfection Gap Factor (PGF), a normalized metric that measures change in performance as a result of task transfer. We utilize PGF to compute Task Transferability, which captures both the breadth and the magnitude of transfer induced by a source task. Using three open-weight VLMs evaluated across 13 perception tasks, we construct a task transfer graph that reveals previously unobserved relationships among perception tasks. Our analysis uncovers patterns of positive and negative transfer, identifies groups of tasks that mutually influence each other, organizes tasks into personas based on their transfer behavior and demonstrates how PGF can guide data selection for more efficient training. These findings highlight both opportunities for positive transfer and risks of negative interference, offering actionable guidance for advancing VLMs.

Related

• Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding
• Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchains
• Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models
• Steering the Verifiability of Multimodal AI Hallucinations
• PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models
• Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding

Source: arXiv cs.CV | 2026-04-10