JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems

arXiv:2603.26515v2 Announce Type: replace-cross Abstract: Efficient and robust turn-taking detection remains a significant challenge in industrial-grade voice AI agents. Existing systems often rely on acoustic or linguistic cues alone, limiting accuracy and stability, whereas directly endowing large language models with full-duplex capability incurs heavy data and deployment costs that hinder real-time usage. We propose JAL-Turn, a lightweight speech-only turn-taking framework based on a joint acoustic-linguistic modeling paradigm, in which a cross-attention module adaptively fuses pre-trained acoustic and linguistic representations for low-latency hold or shift prediction. By sharing a frozen ASR encoder, JAL-Turn enables turn-taking prediction to run fully in parallel with speech recognition, introducing almost no additional end-to-end latency. We further introduce a scalable pipeline that automatically derives reliable turn-taking labels from large-scale real-world dialogue corpora. Experiments on the public Easy-Turn and Smart-Turn benchmarks and an in-house Japanese customer-service dataset show that JAL-Turn achieves competing results among strong baselines in detection accuracy while maintaining superior real-time performance.

Related

• Human-1 by Josh Talks: A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations
• Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework
• Adaptive Turn-Taking for Real-time Multi-Party Voice Agents
• IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systems
• TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue

Source: arXiv cs.AI | 2026-10-15