Abstract

Peer Review in a General Medical Research Journal Before and During the COVID-19 Pandemic

Roy H. Perlis,1,2 Jacob Kendall-Taylor,3 Ishani Ganguli,2,4 Kamber Hart,5 Jesse A. Berlin,2,6 Steven M. Bradley,2,7 Sebastien Haneuse,2,8 Sharon K. Inouye,2,9 Elizabeth A. Jacobs,2,10 Arden Morris,2,11 Eli Perencevich,2,12 Lawrence N. Shulman,2,13 N. Seth Trueger,2,14 Stephan D. Fihn,2,15 Frederick P. Rivara,2,15 Annette Flanagin3

Objective

Although peer review is an essential component of publication for new research, the viability of this process has been questioned, particularly with the added stressors of the COVID-19 pandemic. This study characterized rates of peer reviewer acceptance of invitations to review manuscripts, reviewer turnaround times, and editor-assessed quality of reviews before and after the start of the COVID-19 pandemic at a large, open-access general medical journal.

Design

This retrospective, pre-post cohort study examined all research manuscripts submitted to JAMA Network Open between January 1, 2019, and June 29, 2021, either directly or via transfer from other JAMA Network journals, for which at least 1 peer review of manuscript content was solicited. Measures were compared between the period prior to the World Health Organization declaration of a COVID-19 pandemic on March 11, 2020 (14.3 months), and the period during the pandemic (15.6 months) among all reviewed manuscripts, and between pandemic-period manuscripts that did or did not address COVID-19. For each reviewed manuscript, the number of invitations sent to reviewers, proportions of reviewers accepting invitations, time in days to return reviews, and editor-assessed quality ratings of reviews were determined.

Results

In total, the journal sought review for 5013 manuscripts, including 4295 Original Investigations (85.7%) and 718 Research Letters (14.3%); 1860 manuscripts were submitted during the prepandemic period and 3152 during the pandemic period. Overall mean (SD) volume of manuscripts reviewed per week increased from 30.3 (8.6) to 46.4 (12.2) manuscripts; P < .001. Comparing the prepandemic period with the pandemic period, a greater proportion of invited reviewers declined to review during the pandemic (from 33.0% to 34.5%; P = .02), and the mean (SD) number of reviewer invitations per manuscript increased (from 5.99 [3.57] to 6.99 [4.46]; P < .001). However, the mean (SD) number of reviews rated as high quality (very good or excellent) per manuscript increased from 1.28 (0.72) to 1.48 (0.68), and the mean (SD) time for reviewers to return reviews was modestly shorter (from 15.82 [7.61] days to 14.35 [6.95] days; P < .001), a difference that persisted in regression models accounting for manuscript type and topic.

Conclusions

In this cohort study, peer reviewers were less likely to accept invitations to review manuscripts during the pandemic, but the speed and editor-reported quality of reviews improved. Additional study encompassing a broader set of journals will be necessary to understand the generalizability of these results, and to clarify how the pandemic has affected reviewer burden and fatigue.

1Harvard Medical School and Massachusetts General Hospital, Boston, MA, USA, rperlis@mgh.harvard.edu; 2JAMA Network Open, Chicago, IL, USA; 3JAMA Network, Chicago, IL, USA; 4Harvard Medical School and Brigham and Women’s Hospital, Boston, MA, USA; 5Harvard Medical School, Boston, MA, USA; 6Rutgers School of Public Health, Piscataway, NJ, USA; 7Minneapolis Heart Institute and Minneapolis Heart Institute Foundation, Minneapolis, MN, USA; 8Harvard T. H. Chan School of Public Health, Boston, MA, USA; 9Harvard Medical School, Hebrew SeniorLife, and Beth Israel Deaconess Medical Center, Boston, MA, USA; 10MaineHealth and Maine Medical Center Research Institute, Scarborough, ME, USA; 11Stanford University School of Medicine, Stanford, CA, USA; 12Carver College of Medicine, University of Iowa, Iowa City, IA, USA; 13Abramson Cancer Center, University of Pennsylvania, Philadelphia, PA, USA; 14Department of Emergency Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL, USA; 15University of Washington School of Medicine, Seattle, WA, USA

Conflict of Interest Disclosures

Roy H. Perlis, Ishani Ganguli, Jesse A. Berlin, Steven M. Bradley, Sebastien Haneuse, Sharon K. Inouye, Elizabeth A. Jacobs, Arden Morris, Eli Perencevich, Lawrence N. Schulman, N. Seth Trueger, Stephan D. Fihn, and Frederick P. Rivara report receiving financial support from JAMA Network for service as editors at JAMA Network Open. Jacob Kendall-Taylor and Annette Flanagin are paid editorial staff for the JAMA Network. Annette Flanagin is executive director of the Peer Review Congress but was not involved in the review or decision of this abstract. Roy M. Perlis additionally reports receiving personal fees for service on scientific advisory boards for Belle Artificial Intelligence, Burrage Capital, Circular Genomics, Genomind, Psy Therapeutics, and RID Ventures and holding equity in Belle Artificial Intelligence, Psy Therapeutics, and Circular Genomics.

Poster