Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation

arXiv:2604.17656v1 Announce Type: cross Abstract: Video-to-music (V2M) is the fundamental task of creating background music for an input video. Recent V2M models achieve audiovisual alignment by typically relying on visual conditioning alone and provide limited semantic and stylistic controllability to the end user. In this paper, we present Video-Robin, a novel text-conditioned video-to-music generation model that enables fast, high-quality, semantically aligned music generation for video content. To balance musical fidelity and semantic understanding, Video-Robin integrates autoregressive planning with diffusion-based synthesis. Specifically, an autoregressive module models global structure by semantically aligning visual and textual inputs to produce high-level music latents. These latents are subsequently refined into coherent, high-fidelity music using local Diffusion Transformers. By factoring semantically driven planning into diffusion-based synthesis, Video-Robin enables fine-grained creator control without sacrificing audio realism. Our proposed model outperforms baselines that solely accept video input and additional feature conditioned baselines on both in-distribution and out-of-distribution benchmarks with a 2.21x speed in inference compared to SOTA. We will open-source everything upon paper acceptance.

Related

• Self-Correcting Text-to-Video Generation with Misalignment Detection and Localized Refinement
• ProPhy: Progressive Physical Alignment for Dynamic World Simulation
• Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation

Source: arXiv cs.CL | 2026-04-21