Abstract
A Survey of Authors’ Experiences With Poor Peer Review Practices
Kyle McCloskey,1 Jon F. Merz2
Objective
A constructive peer review can bring a fresh perspective to help improve an author’s work. Until recently, there has been little guidance on what constitutes a good or poor peer review and no empirical study about researchers’ experiences with unhelpful or upsetting reviews.1-3 This pilot study aimed to explore the types of poor peer review practices (PPRP) experienced by authors and assess their association with authors’ ability to disseminate research.
Design
An anonymous 51-question survey was designed through Qualtrics. An invitation to complete the survey was emailed to a random sample of 500 researchers funded by the National Institutes of Health in 2018 (with the replacement of undeliverable email addresses) and posted to a bioethics discussion forum (mcw-bioethics@mailman.mcw.edu) with approximately 600 members. Three mailings were performed between April 5 and April 19, 2022. The study was determined to be exempt by the University of Pennsylvania institutional review board. A 28-item list of PPRP was developed following a literature review; respondents’ experiences were assessed with yes-no and open-ended questions. The PPRP was assessed through author responses to Likert scale questions. Exploratory nonparametric analyses of respondents’ reported experiences and demographic characteristics were performed using Stata, version 12.1 (StataCorp LLC). Data were reported according to the Consensus-Based Checklist for Reporting of Survey Studies (CROSS).
Results
A total of 112 researchers completed the surveys, approximately 10% of those solicited. Respondents were predominantly male (59 of 109 [54%]), held a PhD (87 of 112 [78%]), were older than 50 years (73 of 111 [66%]), identified as White race (92 of 106 [87%]), published more than 50 peer-reviewed papers in their career (71 of 111 [64%]), were trained in the humanities or social sciences (61 of 112 [54%]), and conducted primarily empirical research (68 of 111 [61%]). The mean number of PPRP experienced per author was 12.5 of 28 (44.6%) (range, 0-27; 95% CI, 11.2-13.8). A total of 57% (63 of 111) of authors admitted abandoning a manuscript after receiving unfair peer reviews; 67% (74 of 111) of authors sometimes or often received insightful peer reviews that improved the quality of their final papers. An exploratory univariate analysis of the association between the total number of PPRP reported per author and demographic characteristics and reported positive peer review experiences is presented in Table 62. This analysis suggests areas for future study, particularly the need to explore types of helpful reviews.
Conclusions
This pilot study was the first to date to assess researchers’ experiences with PPRP. The low response rate suggests high selectivity and bias and that respondents report experiences with an extensive range of PPRP. The study lays the groundwork for future research, which may be valuable for improving peer review quality.
References
1. Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006;99(4):178-182. doi:10.1258/jrsm.99.4.178
2. Mavrogenis AF, Quaile A, Scarlat MM. The good, the bad and the rude peer-review. Int Orthop. 2020;44(3):413-415. doi:10.1007/s00264-020-04504-1
3.Silbiger NJ, Stubler AD. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. PeerJ. 2019;7:e8247. doi:10.7717/peerj.8247
1Drexel University College of Medicine, Philadelphia, PA, USA; 2Perelman School of Medicine, University of Pennsylvania, Department of Medical Ethics and Health Policy, Philadelphia, PA, USA, merz@pennmedicine.upenn.edu
Conflict of Interest Disclosures
None reported.