What Do Audio-Visual Synchronization Metrics Actually Measure?

arXiv:2608.25157v1 Announce Type: new Abstract: Automatic AV-sync metrics are widely used to rank and train audio-visual generators, but they are rarely audited as measurement instruments. We jointly audit AV-Align, ImageBind AV-relevance, JavisScore, and Synchformer/DeSync under a common reliability protocol: controlled-distortion monotonicity, preprocessing sensitivity, rank uncertainty, cross-metric agreement, PEAVS-proxy agreement, and learned fusion. The result is an axis split, not a single winner: Synchformer/DeSync is the strongest temporal-offset tracker (au=0.84), ImageBind/JavisScore better match the PEAVS human-aligned proxy (au=0.20) and content-disruption families, and AV-Align is the weakest standalone metric. The metrics mutually disagree (Krippendorff alpha=0.066), and neither linear nor simple k-NN fusion improves PEAVS agreement over the best individual metric. We recommend reporting AV-sync as a Reliability Card (metric-family breakdowns with confidence intervals) rather than a single bare synchronization score.

Related

• GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting
• Modeling Biomechanical Constraint Violations for Language-Agnostic Lip-Sync Deepfake Detection
• Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention
• VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition

Source: arXiv cs.CV | 2026-08-27