Benchmarking Clinical Decision Pathway Adherence in Large Language Models

arXiv:2608.26592v1 Announce Type: new Abstract: Following clinical decision pathways (CDPs) defined by clinical practice guidelines is essential for safe and reliable medical decision-making. However, existing medical large language model (LLM) benchmarks mainly evaluate final-answer accuracy, providing limited evaluation of models' ability to adhere to guidelines. To address this gap, we introduce MEGA-CDP, a benchmark for evaluating whether medical LLMs can generate guideline-adherent CDPs using provided guidelines as references. MEGA-CDP is constructed from 2,274 English and Chinese clinical practice guidelines through a guideline-to-case pipeline, yielding 42,353 clinical cases with explicit reference CDPs. It supports both single-turn vignette and multi-turn interactive settings, and introduces a CDP-oriented evaluation framework for measuring pathway consistency. Experiments on 16 representative LLMs show that reliable clinical decision support remains challenging for current models, demonstrating the need for CDP-oriented evaluation and the value of MEGA-CDP for advancing guideline adherence in medical LLMs.

Related

• Benchmarking and Adapting On-Device LLMs for Clinical Decision Support
• Evaluating Clinical Competencies of Large Language Models with a General Practice Benchmark
• Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases
• Agentic systems for breast cancer treatment recommendations

Source: arXiv cs.CL | 2026-08-28