Safety
FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision
arXiv:2606.14049v2 Announce Type: replace-cross Abstract: We present FoleyGenEx, a unified video-to-audio (VTA) framework integrating multi-modal control, frame-level temporal alignment, and fine-grai
arXiv:2606.14049v2 Announce Type: replace-cross Abstract: We present FoleyGenEx, a unified video-to-audio (VTA) framework integrating multi-modal control, frame-level temporal alignment, and fine-grained semantics, enabling synchronized, versatile audio synthesis for diverse tasks. Existing VTA methods either have multi-modal control but weak temporal alignment or strong alignment but lack reference audio conditioning and semantic precision. FoleyGenEx fills this gap via three core innovations: a conditional injection mechanism for audio-controlled VTA and Foley extension, a multi-modal dynamic masking strategy preserving training synchronization, and an adverb-based data augmentation algorithm leveraging signal processing and large language models to enhance textual supervision with nuanced semantics. Experiments on AudioCaps, VGGSound, and Greatest Hits demonstrate its competitive controllable VTA performance against existing methods. Demo samples are available at https://foleygenex.github.io/FoleyGenEx.
Related
- JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing
- MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation
- OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation
- Here the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation
- Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
Source: arXiv cs.CV | 2026-10-06