Research
PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews
arXiv:2609.11559v1 Announce Type: cross Abstract: Large language models (LLMs) and AI-enabled software increasingly participate in systematic-review decisions, yet the information needed to audit thes
arXiv:2609.11559v1 Announce Type: cross Abstract: Large language models (LLMs) and AI-enabled software increasingly participate in systematic-review decisions, yet the information needed to audit these workflows is reported inconsistently. We analyze SciLitBench, a corpus of 888 review-automation papers with 14,726 annotations, to characterize changes in methods, review-stage use, evaluation and reported limitations. Automation has shifted toward LLM- and software-facing workflows, including stages that can alter the evidence base. Since 2023, 38.0% of software/product papers reported no evaluation, compared with 9.3% of LLM papers. Reporting coverage increased with LLM workflow complexity, yet 52% of positive-only LLM evaluations still reported an unmet reliability or performance requirement. From these patterns, we introduce PRISMA-LLM, an empirically grounded framework separating implementation disclosure from consequence-sensitive evaluation and limitation reporting.
Related
- Not What, But How: A Communicative Audit of LLM Response Framing
- Do large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effects
- From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation
- The Score Granularity Gap in Black-Box LLM Classification: A Comparative Study of Confidence Constructions
Source: arXiv cs.CL | 2026-09-14