Abstract
Unprofessional Comments in Peer Review Reports Across Scholarly Disciplines
Mario Malički,1 Taym Alsalti,2 Daniel García-Costa,3 Francisco Grimaldo,3 Elena Álvarez-García,3 Ana Jerončić,4 Steven M. Goodman,5 Flaminio Squazzoni,6 Bahar Mehmani7
Objective
Previous research has indicated that 642 of 1116 surveyed researchers (58%) received unprofessional comments in peer review reports at least once in their professional life1 and that 179 of 1491 review reports (12%) in the fields of ecology and evolution and behavioral medicine contained unprofessional comments.2 It was the goal of this study to estimate the percentage of manuscripts with at least 1 unprofessional comment made in a broad sample of peer review reports across disciplines.
Design
This was a cross-sectional study of review reports available in the PEERE database, which covers 10 years of peer review records across all impact factor quartiles and peer review types and includes journals from the fields of life sciences, health and medicine, physical sciences, and social sciences and economics.3 Sample size calculation indicated that 380 of 297,026 manuscripts contained in the PEERE database should be analyzed to detect a prevalence of 1% of manuscripts (range, 0%-2%) having at least 1 unprofessional comment. Randomized stratified sampling was used to preserve scholarly field distribution. All review reports of 1147 sampled manuscripts were then extracted. Reading of the reports was done independently by 3 researchers, who marked instances of unprofessional comments. Disagreement between researchers was resolved through consensus.
Results
Of 1147 analyzed review reports, 13 (1.1%) contained at least 1 unprofessional comment (Table 69). All were found in review reports of different manuscripts; authors of 13 of 380 analyzed manuscripts (3.4%) received unprofessional comment(s) during peer review. All instances of unprofessional comments were found among detailed review reports.
Conclusions
In the study sample, 3.4% of submitted manuscripts received at least 1 unprofessional comment during peer review. Although the PEERE database presents the largest collection of confidential review reports shared by publishers, it might not be representative of all journals. Nevertheless, to the best of our knowledge, this is the first study of unprofessional comments that used random sampling across scholarly disciplines. Future studies should evaluate the effects of these comments on the authors, assess whether and how they respond to them, and explore automated approaches for detecting unprofessional comments.
References
1. Silbiger NJ, Stubler AD. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. Peer J. 2019;7:e8247. doi:10.7717/peerj.8247
2. Gerwing TG, Gerwing AMA, Avery-Gomm S, Choi CY, Clements JC, Rash JA. Quantifying professionalism in peer review. Res Integ Peer Rev. 2020;5(1):1-8. doi:10.1186/s41073-020-00096-x
3. Squazzoni F, Bravo G, Farjam M, Marusic A, Mehmani B, Willis M, et al. Peer review and gender bias: a study on 145 scholarly journals. Sci Advanc. 2021;7(2):eabd0299. doi:10.1126/sciadv.abd0299
1Meta-Research Innovation Center at Stanford, Stanford University, Stanford, CA, USA, mario.malicki@mefst.hr; 2Personality Psychology and Psychological Diagnostics, University of Leipzig, Leipzig, Germany; 3Department of Computer Science, University of Valencia, Burjassot-Valencia, Spain; 4Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia; 5Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, USA; 6Department of Social and Political Sciences, University of Milan, Milan, Italy; 7STM Journals, Elsevier, Amsterdam, the Netherlands
Conflict of Interest Disclosures
Mario Malički is the co–editor in chief of Research Integrity and Peer Review. Bahar Mehmani is reviewer experience lead at Elsevier. Other authors declare no competing interests. Steven Goodman and Bahar Mehmani are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.
Funding/Support
The work by Francisco Grimaldo, Daniel García-Costa, and Elena Álvarez-García is supported by the Spanish Ministry of Science, Innovation and Universities, the Spanish State Research Agency, and the European Regional Development Fund under project RTI2018-095820-B-I00.
Role of Funder/Sponsor
The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.