TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models

arXiv:2609.01277v1 Announce Type: cross Abstract: Although pretrained joint audio-visual diffusion models offer rich control over what to generate, they provide no explicit control over when an utterance should occur. To address this, we study inference-time speech scheduling, a novel task that places coupled speech and visual articulation within user-specified begin--end intervals without finetuning the backbone model. We uncover two intrinsic properties of the denoising process that enable this task. First, a timing-sensitive text-to-audio cross-attention head exposes each utterance's model-implied source span along the latent timeline. Second, the predicted clean latent already organizes coupled speech and visual articulation, allowing their temporal placement to be edited without regenerating the content. Building on these discoveries, we propose extbf{TimeSteer}, a training-free framework that localizes each utterance's source span through extbf{Source Span Localization} and transfers the associated audio-visual latent content from the source interval to the specified target interval through extbf{Region-Aware Latent Remapping}. We further introduce extbf{SpeechShift}, the first benchmark for interval-level speech scheduling in joint audio-visual generation. Experiments across two representative backbones show that TimeSteer substantially improves interval controllability over training-free baselines while maintaining competitive overall generation quality.

Related

• Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models
• dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model

Source: arXiv cs.AI | 2026-09-02