Drop the Details, Get the Gist: Dynamic Conceptual Representation for Video Recognition

arXiv:2610.36832v1 Announce Type: new Abstract: Concept Bottleneck Models (CBMs), which route predictions through a layer of human-interpretable concepts, have substantially improved the interpretability of visual classification by exposing an explicit, language-based rationale for each prediction. Most existing work focuses on static images, and temporal dynamics in videos have been underexplored in conceptual representation. We propose a framework that extends concept-based image recognition to procedural video understanding. It represents a multi-step video as a temporal sequence of segment-level concept score vectors and feeds this sequence into a Transformer encoder to predict the class label. We also explore a way to estimate the contribution of each concept for further interpretability. We evaluated three concept sets of different granularities on cooking and household-activity videos and found that coarser-grained concepts are sufficient for classifying these videos, which implies that fine-grained textual descriptions may be unnecessary.

Related

• Concepts in Motion: Temporal Concept Bottleneck Model for Interpretable Video Classification
• Beyond Heatmaps: Unsupervised Concept-Graph Reasoning for Interpretable Visual Explanation
• SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

Source: arXiv cs.CV | 2026-10-28