Talk in Pieces, See in Whole: Disentangled and Hierarchical Representation Learning in Language-based Object Detection

arXiv:2509.24192v2 Announce Type: replace Abstract: Vision-language models (VLMs) have advanced multimodal perception, demonstrated by open-vocabulary object detection with simple language queries. State-of-the-art VLMs still struggle to handle complex queries involving descriptive attributes and relational clauses. To address this problem, we propose restructuring linguistic representations according to the hierarchical relations within sentences for language-based object detection. A key insight is that textual tokens should be disentangled into core components-objects, attributes, and relations-and aggregated into hierarchically structured sentence-level representations. Building on this principle, we introduce the TaSe (Talk in Pieces, See in Whole) framework with three main contributions: (1) a hierarchical synthetic captioning dataset spanning three tiers from category names to descriptive sentences; (2) the three-component disentanglement module guided by a novel disentanglement loss function, transforms text embeddings into subspace compositions; and (3) aggregating disentangled components into hierarchically structured embeddings guided by the proposed hierarchical objectives. Experimental results under the OmniLabel benchmark show a 24% performance improvement, demonstrating the importance of linguistic compositionality.

Related

• Fine-Grained Open-Vocabulary Object Detection with Fined-Grained Prompts: Task, Dataset and Benchmark
• VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio
• FlowOVD: Learning Generative Latent Flows for Zero-shot Open-vocabulary Detection
• A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions

Source: arXiv cs.CV | 2026-08-31