Research
Structured-Noise Masked Modeling for Video, Audio and Beyond
arXiv:2503.16311v2 Announce Type: replace-cross Abstract: Masked modeling has emerged as a robust self-supervised learning framework. However, most methods rely on random masking, which disregards the
arXiv:2503.16311v2 Announce Type: replace-cross Abstract: Masked modeling has emerged as a robust self-supervised learning framework. However, most methods rely on random masking, which disregards the structural properties of different data modalities. To align with the spatiotemporal and spectral characteristics of video and audio data, we introduce a structured noise-based masking approach. By filtering white noise into different color noise distributions, we generate structured masks that capture modality-specific patterns without requiring handcrafted heuristics or access to the data. Our approach enhances masked video and audio modeling frameworks without any additional computational cost. Experiments show that structured noise masking consistently outperforms random masking, underscoring the value of modality-aware masking strategies for representation learning.
Related
- Mitigating Positional Leakage in 3D Masked Autoencoders for Robust Representation Learning
- Typhoon: Towards an Effective Task-Specific Masking Strategy for Pre-trained Language Models
- Generative Modeling from Black-box Corruptions via Self-Consistent Stochastic Interpolants
Source: arXiv cs.AI | 2026-10-02