Safety
Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
arXiv:2608.18671v1 Announce Type: new Abstract: Egocentric video captures activities from the wearer's perspective, providing a direct view of human attention, hand--object interaction, and goal-direc
arXiv:2608.18671v1 Announce Type: new Abstract: Egocentric video captures activities from the wearer's perspective, providing a direct view of human attention, hand--object interaction, and goal-directed behavior. This perspective is increasingly important for wearable intelligence, assistive systems, human--robot interaction, and embodied AI, yet it introduces challenges including ego-motion, occlusion, small active objects, viewpoint-dependent appearance, and long-range temporal dependencies. Vision--language models (VLMs) offer a promising foundation for addressing these challenges by linking visual observations with semantic knowledge and natural-language supervision. This survey presents a critical review of VLMs for egocentric video understanding, tracing the progression from conventional recognition architectures to multimodal foundation models and embodied systems. We organize the literature around tasks, datasets, hand--object interaction understanding, temporal reasoning, frame and clip selection, multimodal representation learning, prompting, semantic alignment, and model adaptation. Particular attention is given to graph-based and object-centric reasoning as mechanisms for modeling relations among hands, objects, actions, and scene context over time. We further examine how first-person perception and multimodal foundation models support wearable assistance, robot skill learning, human-to-robot transfer, and embodied decision making. Across the reviewed literature, a consistent limitation emerges: current models recognize visible objects more reliably than evolving interactions, actions, and user intent, especially over long activities. We therefore identify temporally grounded reasoning, interaction-aware supervision, efficient long-video processing, multimodal fusion, graph-enhanced representations, cross-domain generalization, privacy, and trustworthy evaluation as key priorities for deployable embodied intelligence.
Related
- DextER: Language-driven Dexterous Grasp Generation with Embodied Reasoning
- Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models
- FROST-STA: Frozen Dense Features for the Ego4D Short-Term Object Interaction Anticipation
- Improving Reasoning in Vision-Language Models via Perception Verified Self-Training
Source: arXiv cs.CV | 2026-08-20