DSDyn-VLA: A Dual-Stream Dynamic Manipulation Framework with Motion Perception, Future Awareness, and Realtime Correction

arXiv:2609.39198v1 Announce Type: new Abstract: While Vision-Language-Action (VLA) models excel in static tasks, they struggle in dynamic environments where objects are in motion (e.g., conveyor belt manipulation). We identify three fundamental limitations hindering current VLAs in these scenarios: the extbf{perception gap}, where static visual inputs lack temporal motion cues; the extbf{latency gap}, where inference delays render actions obsolete; and the extbf{control gap}, caused by the open-loop action chunk execution without real-time adjustment. In this work, we propose extbf{DSDyn-VLA}, a Slow-Fast extbf{D}ual-extbf{S}tream extbf{Dyn}amic manipulation framework that integrates motion-aware foresighted planning with real-time residual correction. The slow extbf{Flow-Planner} serves as a macro-planner. By enhancing the VLA with optical flow for temporal perception and a future state awareness mechanism to preemptively offset inference latency, it produces globally consistent, motion-aware action chunks. Complementing this, the fast extbf{Res-Refiner} employs a lightweight RL policy to inject high-frequency, closed-loop corrections into the planned action chunks based on real-time observations. In addition, we introduce extbf{DynBench}, a MuJoCo-based benchmark for dynamic object manipulation that comprises nine tasks. Extensive experiments demonstrate that DSDyn-VLA reduces the failure rate by over 76% compared to current SOTA method in high-latency setting on the Kinetix dynamic benchmark, while achieving about 6imes the success rate of PI0.5 in real-world dynamic settings and about 5imes on DynBench. We will open-source all the code and weights.

Source: arXiv cs.RO | 2026-10-01