An AI-Assisted Analysis of Published PeerJ Open Peer Reviews

Peiling Wang,¹ Dietmar Wolfram,² Scott Shumate³

Objective

Open peer review processes promote transparency and accountability. As more publishers adopt open peer review,¹ accessible peer reviews become valuable corpora to develop AI tools for enhancing peer review processes.² The main purpose of peer review as stated in the Guidelines for Reviewers³ is, “First, reviews provide constructive advice and recommendations to the authors on how their paper can be improved.” How can we determine the constructiveness of peer review reports? This exploratory study analyzed 11,321 peer review reports of the first version of 4100 manuscripts submitted to, and published in, PeerJ to identify constructive comments. Specifically, this dataset includes first reviews from November 11, 2012, to November 24, 2019. This study examines 3 research questions: (1) To what extent do reviewers provide constructive comments? (2) To what extent do reviewers use hedging to soften criticisms or recommendations? (3) Are there differences in constructive comments between signed and anonymous reviews?

Design

The gpt-4o-2024-08-06 model from OpenAI API was used to analyze reviews at the sentence level to identify criticisms or supportive comments about different aspects of the research, including study purpose, hypotheses, methods, results, discussion, and conclusions. A detailed system prompt (https://zenodo.org/records/15464322) was used with a temperature setting of 0. The model was instructed to categorize each sentence into a structured JSON output specifying section, aspects, evaluation, tone, and mode. Use of hedging tone includes phrases such as “I do not think the investigation was ‘rigorous’” or “has the potential to be much better and more relevant.” Building on the GPT analysis, we derived the following variables to measure review feature aspects, including (1) Pc, the percentage of critical sentences; (2) Ps, the percentage of supportive sentences; and (3) Ph, the percentage of hedging sentences. The constructive index (CI), considering both criticisms and supportive comments, is the percentage of sentences containing research comments.

Results

Of the 11,321 peer review reports, 7071 (62.5%) were anonymous and 4250 (37.5%) were signed (Table 25-1055). Few reports were supportive (overall median of 9%). Higher levels (27% of anonymous reviews and 25% of signed reviews) contained criticisms of the research. A similar level of hedging was observed. Significant differences were observed between the 2 groups; signed reviews showed significantly fewer criticisms and higher levels of hedging.

Conclusions

We measured the extent of research-related comments in peer review reports that authors opted to publish. Anonymous reviewers tend to be more critical. The use of hedging in reviews could obscure clear suggestions on how to improve the manuscript. This research represents an initial effort towards a model for an AI-assisted peer review system. Further analysis should fine-tune the AI algorithms for analysis of reviewer comments. In addition, authors’ perspectives on what constitutes constructive feedback warrant further study. This research had limitations. Review histories were not available for rejected manuscripts or for articles where authors opted out of publishing reviews.

References

1. Wolfram D, Wang P, Hembree A, Park H. Open peer review: promoting transparency in open science. Scientometrics. 2020;125:1033-1051. doi:10.1007/s11192-020-03488-4

2. Wolfram D, Wang P, Abuzahra F. An exploration of referees’ comments published in open peer review journals: the characteristics of review language and the association between review scrutiny and citations. Res Eval. 2021;30(3):314-322. doi:10.1093/reseval/rvab005

3. Technical Community on Real-Time Systems (TCRTS) of the IEEE Computer Society. Guidelines for reviewers. Accessed on May 29, 2025. https://cmte.ieee.org/tcrts/guidelines-for-reviewers

¹University of Tennessee, Knoxville, US, peilingw@utk.edu; ²University of Wisconsin, Milwaukee, US; ³Austin Peay State University, Clarksville, TN, US.

Conflict of Interest Disclosures

None reported.