Topic Matching in the Wild: Benchmark and Lessons from Real-World ASR Transcripts

arXiv:2609.00330v1 Announce Type: cross Abstract: In contact centers, real-time agent-assist tools determine, for each of many predefined topics, whether a live customer utterance is relevant and display a coaching card to the agent when it is. The input is noisy and challenging: ASR(Automatic Speech Recognition) transcripts of spontaneous phone conversations, which can be unclear, repetitive, and mostly lack punctuation. To systematically study this real-world task, we curate a human-annotated topic-utterance judgments dataset sourced from real call-center transcripts. We compare three types of matchers: a regex-based baseline, zero-shot sentence embedding encoders, and Gemini-based LLM matchers. In addition, two types of topic representations are studied in our benchmark:keyphrases and natural language description. Our empirical experiments highlight the superior performance of lightweight LLM matchers over embedding and regex models when equipped with natural language descriptions.

Related

• Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
• A Benchmark for Multi-Party Negotiation Games from Real Negotiation Data
• Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

Source: arXiv cs.AI | 2026-09-02