Visual Prompting for Robotic Manipulation with Annotation-Guided Pick-and-Place Using ACT

arXiv:2508.08748v2 Announce Type: replace-cross Abstract: Robotic pick-and-place tasks in convenience stores pose challenges due to dense object arrangements, occlusions, and variations in object properties such as color, shape, size, and texture. These factors complicate trajectory planning and grasping. This paper introduces a perception-action pipeline leveraging annotation-guided visual prompting, where bounding box annotations identify both pickable objects and placement locations, providing structured spatial guidance. Instead of traditional step-by-step planning, we employ Action Chunking with Transformers (ACT) as an imitation learning algorithm, enabling the robotic arm to predict chunked action sequences from human demonstrations. This facilitates smooth, adaptive, and data-driven pick-and-place operations. We evaluate our system based on success rate and visual analysis of grasping behavior, demonstrating improved grasp accuracy and adaptability in retail environments.

Related

• PhysMem: Scaling Test-time Physical Memory for Robot Manipulation
• Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes
• NebulaVLA: A Dual-Frequency Vision-Language-Action Model With Guide Action for Robotic Manipulation
• GaussMemory: Task-Driven 3D Gaussian Scene Memory for Long-Horizon Robotic Manipulation

Source: arXiv cs.AI | 2026-08-19