Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation

arXiv:2608.29121v1 Announce Type: new Abstract: Open-Vocabulary Audio-Visual Semantic Segmentation (OV-AVSS) aims to perform pixel-level segmentation of sound-emitting objects from an open set of categories. The previous method relies on a class-agnostic foreground definition, which groups semantically diverse objects into a heterogeneous positive set, causing the model to learn unstable sounding patterns and produce unreliable proposals. To address this, we reformulate the objective to be category-specific and propose a novel Acoustically Grounded Cost Learning (AGCL) framework to transform the static, audio-agnostic visual-text priors into dynamic, audio-grounded cost representations. For intra-category soundingness discovery, we devise Audio-Modulated Cost Generation (AMCG) and Audio-Guided Temporal Aggregation (AGTA) modules to enable both frame-level sounding region highlighting and video-level temporal refinement with a low-intrusive audio injection mechanism. For inter-category distractor discrimination, we introduce a Synergistic Distractor Mining (SDM) strategy, which selectively penalizes acoustically and semantically confusing negative categories to learn more discriminative decision boundaries. Extensive experiments on the AVSBench-OV dataset demonstrate that our method significantly outperforms previous state-of-the-art approaches, particularly on unseen categories. Code is available at https://github.com/spyflying/AGCL.

Related

• C^3ASD: Multi-Level Consistency-Driven Representation Learning
• Listen, Look, and Learn: Learning Without Forgetting through SAM-Audio
• Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understanding
• MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning

Source: arXiv cs.CV | 2026-09-01