Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance

arXiv:2602.10118v2 Announce Type: replace Abstract: Peer review is central to scientific quality, yet reliance on simple heuristics, namely lazy thinking and non-specific critiques, has threatened review quality. Prior work frames lazy thinking detection as single-label classification and stops at detection, yet review segments often exhibit multiple co-occurring issues, and reviewers benefit more from actionable, guideline-aware feedback than from labels alone. We further show that off-the-shelf LLMs prompted for feedback frequently rewrite the entire review or address the authors rather than the reviewer, motivating an inference-time approach. We introduce an LLM-driven framework that decomposes reviews into argumentative segments, identifies issues violating ACL Rolling Review (ARR) guidelines, and generates targeted feedback using issue-specific templates refined by a novel iterative, reranking-based generation algorithm. In a controlled rewriting study, our feedback reduces guideline violations by up to 92.4%. We also release LazyReviewPlus, the first multi-label dataset of 1,309 sentences annotated for detecting lazy thinking and lack of specificity.

Related

• ReviewGuard: Aligning LLM-Assisted Peer Review with Long-Term Scientific Impact
• Decoupling Scores and Text: The Politeness Principle in Peer Review
• PeeriScope: A Multi-Faceted Framework for Evaluating Peer Review Quality
• Aspect-Based Sentiment Evolution and its Correlation with Review Rounds in Multi-Round Peer Reviews: A Deep Learning Approach

Source: arXiv cs.CL | 2026-09-03