Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval

arXiv:2604.23692v1 Announce Type: cross Abstract: Audio-driven facial animation is essential for immersive digital interaction, yet existing frameworks fail to reconcile real-time streaming with high-fidelity personalization. Current methods often rely on latency-inducing audio look-ahead, or require high user compliance to pre-encode static embeddings that fails to capture dynamic idiosyncrasies. We present an end-to-end causal framework for personalizing causal facial motion generation via dynamic multi-modal style retrieval, enabling ultra-low latency while uniquely leveraging unstructured style references. We introduce two key innovations: (1) a temporal hierarchical motion representation that captures global temporal context and high-frequency details while maintaining decoding causality, and (2) a multi-modal style retriever that jointly queries audio and motion to dynamically extract stylistic priors without breaking causality. This mechanism allows for scalable personalization with total flexibility regarding the number and contents of templates. By integrating these components into a causal autoregressive architecture, our method significantly outperforms state-of-the-art approaches in lip-sync accuracy, identity consistency, and perceived realism, supported by extensive quantitative evaluations and user studies.

Related

• PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination
• JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation
• Real-Time Human Reconstruction and Animation using Feed-Forward Gaussian Splatting
• ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis

Source: arXiv cs.CV | 2026-04-28