Research
More Than Mimicking Reviewers: Evaluating LLMs for Pre-Submission Peer Review
arXiv:2609.05788v1 Announce Type: new Abstract: Peer-review feedback often arrives too late for authors to make meaningful revisions. We study an author-facing LLM system that moves part of this stres
arXiv:2609.05788v1 Announce Type: new Abstract: Peer-review feedback often arrives too late for authors to make meaningful revisions. We study an author-facing LLM system that moves part of this stress test before submission: it generates a broad pool of atomic concerns and compresses them into a short report. We evaluate agreement with historical reviews and, separately, the possible validity of concerns they omit. From 10,000 ICLR 2026 submissions, we use 3,398 manuscripts with accessible versions that predate review. On a ten-paper diagnostic, independent sampling covers 44.9% of historical issues; deduplication and refill reaches 78.7% strict and 84.9% seriousness-weighted coverage, at 3.6imes more requests and 5.2imes more tokens. A hidden Top-32 Oracle preserves the full 79.3% weighted coverage of a 256-candidate pool, but paper-only selectors retain only 40--44%. LLM review therefore provides broad coverage with a large candidate pool but compresses poorly; ablations identify representative selection and matcher sensitivity as the main sources of this gap.
Related
- Do large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effects
- Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Review
- To Trust or Not to Trust: Authors' Response to AI-based Reviews
- Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI
- How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
Source: arXiv cs.AI | 2026-09-10