An overview of 3D Vision-Language Models

arXiv:2609.05583v1 Announce Type: new Abstract: Vision-Language Models (VLMs) are reshaping computer vision by aligning visual and textual embeddings, allowing models to recognize visual concepts and reason about them using natural language. Traditional 3D deep-learning models, however, are typically trained for specific tasks, such as classification, segmentation, or detection, and do not naturally support cross-modal retrieval from their embedding spaces using text or images as queries. To address this issue, Contrastive Language-Image Pretraining (CLIP)-based methods align 3D embeddings with pretrained image and text representations, giving rise to 3D Vision-Language Models (3D VLMs) that support zero-shot classification, cross-modal retrieval, and open-vocabulary recognition of 3D shapes. This tutorial provides an overview of 3D VLMs, ranging from basic definitions of 3D representations and their encoding into embeddings to cross-modal contrastive alignment, modern multimodal frameworks, and 3D Vision-Large Language Models (3D VLLMs). We present the main definitions of contrastive learning for multimodal embedding alignment and highlight recent advances in language-guided 3D Gaussian splatting, 3D shape generation, and embodied AI for robotics.

Related

• Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models
• Hi-Token: Hierarchical Coordinate Tokenization for Generative Visual Grounding
• GAP3D: Generative Alignment of VLM Latents to Patch-Level Embeddings for 3D Generation

Source: arXiv cs.CV | 2026-09-09