TSMD: Temporal-Stream Modality Dropout for Robust Video Highlight Detection

arXiv:2609.39051v1 Announce Type: new Abstract: Existing multimodal video highlight detectors typically assume that visual, audio, and textual streams are continuously available. In practice, however, inputs may suffer from localized frame missingness or complete-stream outage. We formulate this robustness challenge along two dimensions: temporal missingness, where frames are missing independently in each modality, and stream-level missingness, where one modality is unavailable throughout a video. Moreover, we find that the mean squared error (MSE) loss is misaligned with both the evaluation metrics and the peak-driven nature of highlights. Therefore, we propose Temporal-Stream Modality Dropout (TSMD), which combines structured missingness simulation with a joint objective comprising pointwise MSE, per-video Pearson correlation, and peak-oriented RankNet loss terms. TSMD has three variants: temporal, stream-level, and mixed dropout. On the MoSu and Mr. HiSum datasets, TSMD-Temporal improves mAP@15 by 7.06 and 3.41 points over TripleSumm under 50% independent temporal removal, whereas TSMD-Stream performs the best under complete-stream removal. TSMD-Mix retains most of these complementary benefits and ranks the best or the second-best across the evaluated temporal and stream-level conditions.

Related

• Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
• TD-VAD: Breaking Visual Dependence in Video Anomaly Detection with Text-Driven Learning
• EviDETR: Preserving Query-Relevant Temporal Evidence for Moment Retrieval and Highlight Detection

Source: arXiv cs.CV | 2026-10-01