J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

arXiv:2610.19638v1 Announce Type: new Abstract: We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While tau-bench is widely used to evaluate LLM agents, it is English-centric and cannot evaluate agents in other languages. Existing multilingual extensions offer no public Japanese version and rely primarily on machine translation, leaving locale-specific entities such as addresses and currencies inconsistent with the target region. To enable reliable evaluation in Japanese, we carefully localize tau-bench through machine translation, entity adaptation, and human review. During this process, we also uncover and fix two flaws in the original tau-bench: invalid tasks and a score aggregation error that inflates certain models' results. Experiments show that J-tau yields fewer evaluation errors than a naively machine-translated counterpart and that our fixes resolve both flaws.

Related

• Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills
• Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback
• When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue
• OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills

Source: arXiv cs.CL | 2026-10-15