Bootstrapping Video Interaction Generation with Synthetic State Transitions

arXiv:2610.01039v1 Announce Type: new Abstract: While recent video generative models can synthesize high-fidelity videos, they struggle to portray plausible physical interactions and the resulting state transitions, a critical bottleneck for applications in robotics and VR/AR. To address this, we introduce a framework to generate a scalable synthetic dataset of controllable interactions. Our pipeline leverages a structured taxonomy and state-of-the-art image editing models to create explicit start' and end' state images, which serve as visual anchors for the interaction. To generate a seamless video utilizing these anchors, we propose State-Guided Sampling (SGS), a novel sampling technique that mitigates artifacts common in naive conditional generation. Furthermore, we develop and validate a new automated evaluation system that aligns with human judgments to ensure data quality. Experiments show that fine-tuning a base model on our dataset significantly enhances its ability to generate plausible interactions.

Related

• Gen2Physics: Grounding Generated 3D Meshes in Physics via Multi-View Material Decomposition
• Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors
• Physically Plausible Video Generation via Visual-Semantic Chain-of-Events Conditioning
• Proprio: Latent Self-Scoring and Inference-Time Refinement for Physically Plausible Video Generation
• SparseCtrl-HOI: Sparse Temporal Control for Human-Object Interaction Video Generation

Source: arXiv cs.CV | 2026-10-02