Soundwich: Video Generation with Layered and Controllable Audio

arXiv:2610.00691v1 Announce Type: new Abstract: Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed track. This limits source-level control and differs from practical audiovisual workflows, where speech, music, sound effects, and ambient sounds are represented as separate editable tracks. We introduce Soundwich, a training-free framework that transforms a frozen joint audio-video flow-matching model into a generator of multiple synchronized, independently editable audio stems coupled to a shared video. Soundwich generates separate audio stems with explicit control over their temporal activity. To keep separately generated sounds coherent, we introduce a shared scene representation that communicates global audiovisual context across stems while preserving their source-level separation. We further route cross-modal interactions between each audio stem and its corresponding visual source, improving audiovisual consistency. The resulting stems remain synchronized with the video and can be independently retimed, muted, replaced, or remixed. Experiments and human evaluations show improved temporal control, source separation, and naturalness, while enabling flexible source-level editing within coherent audiovisual generation. Code is available at https://github.com/CodyNing/Soundwich.

Related

• Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention
• InteractiveAvatar: Real-Time Streaming Video Generation for Consistent and Intent-Aware Avatars
• Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy
• Mamba-Enhanced Implicit Motion Learning for Audio-Driven Portrait Animation

Source: arXiv cs.CV | 2026-10-02