Model Releases
A multicenter benchmark and clinically structured metric for coronary CTA report generation
arXiv:2609.00909v1 Announce Type: new Abstract: Reliable evaluation of automated coronary computed tomography angiography (CCTA) report generation requires standardized multicentre benchmarks and clin
arXiv:2609.00909v1 Announce Type: new Abstract: Reliable evaluation of automated coronary computed tomography angiography (CCTA) report generation requires standardized multicentre benchmarks and clinically structured metrics. We established a four-centre benchmark comprising 3,021 CCTA series from 818 patient-report pairs to evaluate seven open-source three-dimensional vision-language models. We developed CSM_{ext{CCTA}}, a clinically structured metric for CCTA report evaluation, with patient-, vessel-, and segment-level variables defined according to clinical guidelines. Report pairs are compared at the finest shared anatomical level, and the contributions of different clinical components are weighted based on expert assessments. We estimated these weights using 70 expert-scored cases and evaluated clinical alignment in a non-overlapping set of 30 cases. CSM_{ext{CCTA}} showed a strong correlation with radiologist scores (Pearson's r=0.97, p<0.001), exceeding the next-best metric, FORTE (r=0.70), by 0.27, and agreed with expert preferences in 115 of 160 pairwise comparisons (71.9%). Under controlled perturbations, CSM_{ext{CCTA}} remained stable to clinically equivalent wording and decreased monotonically with progressive information omission. In the multicenter benchmark, the CCTA-trained C2RG model achieved the highest CSM_{ext{CCTA}} scores across all four hospitals, although its performance remained far from optimal. In contrast, CCTA-irrelevant reports accounted for up to 98.7% of the outputs from generalist models. Together, the benchmark provides a standardized setting for model comparison, while CSM_{ext{CCTA}} enables clinically structured evaluation of finding agreement and anatomical specificity. These results support a more clinically aligned and anatomically resolved approach to evaluating CCTA report generation. Code is available at https://openi.pcl.ac.cn/OpenMedIA/CSM_CCTA.
Related
- DobicVLM: Aligning Chest X-Ray Report Generation with Clinically-Grounded Programmatic Rewards via Group Relative Policy Optimization
- Beyond Fluency: A Clinical Benchmark and Anomaly-Enhanced Baseline for Spine MRI Report Generation
- LLaMA-XR: A Novel Framework for Radiology Report Generation using LLaMA and QLoRA Fine Tuning
Source: arXiv cs.CV | 2026-09-02