Enabling a Unified Cross-Domain Representation for Two-Finger Gripper Manipulation via Interaction-Centric Modeling

arXiv:2609.31207v1 Announce Type: cross Abstract: Achieving robust cross-embodiment generalization in imitation learning demands overcoming a critical representation flaw that inextricably entangles task semantics with hardware-specific visual geometry. We propose an interaction-centric framework that leverages the shared structure of two-finger grippers via a parameterized universal gripper abstraction, yielding a canonical gripper-frame representation. Given language and RGB-D observations, a VLM infers the subtask and grounds an interaction triplet (gripper, held, target), while SAM~2.1 tracks masks to reduce VLM queries. We design concise hybrid features that combine target/collision artificial potential fields for global guidance with segmented gripper-frame point clouds for local geometry, and use a Flow-Matching Transformer to predict smooth 7-DoF action chunks. Experiments in simulation and real-world tasks demonstrate that ours is the first imitation learning approach to simultaneously achieve competitive benchmark scores and extreme cross-embodiment/cross-viewpoint zero-shot sim-to-real transfer to completely distinct, heterogeneous robot platforms.

Related

• Cross-Embodiment Transfer via Behavior-Aligned Representations
• VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation
• DynamicManip: Enabling Dynamic Manipulation from a Single Static Demonstration
• SlotVLA: Towards Modeling of Object-Relation Representations in Robotic Manipulation
• RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning

Source: arXiv cs.CV | 2026-09-28