Model Releases
Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition
arXiv:2610.15891v1 Announce Type: new Abstract: Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features
arXiv:2610.15891v1 Announce Type: new Abstract: Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of 75.23%pm0.60 averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.
Related
- Multilevel neural networks with dual-stage feature fusion for human activity recognition
- DAP-Pose: Deep Temporal Alignment and Physics-aware Cross-modal Sensor Fusion for Robust Pose Estimation
- Triple Spectral Fusion for Sensor-based Human Activity Recognition
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness
Source: arXiv cs.CV | 2026-10-14