Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration

arXiv:2604.22491v1 Announce Type: cross Abstract: Selecting out-of-reach objects is a fundamental task in mixed reality (MR). Existing methods rely on a single cue or deterministically fuse multiple cues, leading to performance degradation when the dominant cue becomes unreliable. In this work, we introduce a probabilistic cue integration framework that enables flexible combination of multiple user-generated cues for intent inference. Inspired by natural grasping behavior, we instantiate the framework with pointing direction and grasp gestures as a new interaction technique, Point&Grasp. To this end, we collect the Out-of-Reach Grasping (ORG) dataset to train a robust likelihood model of the gestural cue, which captures grasping patterns not present in existing in-reach datasets. User studies demonstrate that our selection method with cue integration not only improves accuracy and speed over single-cue baselines, but also remains practically effective compared to state-of-the-art methods across various sources of ambiguity. The dataset and code are available at https://github.com/drlxj/point-and-grasp.

Related

• A Kinematic Framework for Evaluating Pinch Configurations in Robotic Hand Design without Object or Contact Models
• Fast-SegSim: Real-Time Open-Vocabulary Segmentation for Robotics in Simulation
• DART: Learning-Enhanced Model Predictive Control for Dual-Arm Non-Prehensile Manipulation
• Novel Algorithms for Smoothly Differentiable and Efficiently Vectorizable Contact Manifold Construction

Source: arXiv cs.RO | 2026-04-27