Abstract
Online Training in Scholarly Peer Review: A Systematic Review
Jess V. Willis,1,2 Janina Ramos,1,3 Ryan Chow,1,2 Mohsen Alayche,1,2 Jeremy Y. Ng,1 Kelly D. Cobey,1,4 David Moher1,5
Objective
To perform a systematic review of available online training for scholarly peer review of biomedical journal articles.
Design
A search strategy was developed and reviewed using the PRESS (Peer Review of Electronic Search Strategies) checklist by a medical librarian. A database search of MEDLINE, PsycINFO, Embase, ERIC, and Web of Science was conducted. Additional supplementary searches were done of preprint servers, Google, YouTube, university library websites, publisher websites, and peer review–related events and groups. All English or French training documents for scholarly peer review of biomedical manuscripts freely accessible online between January 1, 2012, and the date of the search (September 13, 2021) were included. January 1, 2012, was used as the earliest cutoff because this was the year Publons was launched. A Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram with full exclusion criteria is shown in Figure 11. Screening was done in duplicate by 2 independent reviewers in 2 stages, with conflicts resolved by a third party. Data extraction and risk of bias were done by 1 reviewer and then verified by a second. As no current risk of bias tool could be found for evaluating training material, one was created, which was pilot tested for feasibility.
Results
Of 1244 records screened, 45 online training documents were identified for peer review. Barriers such as paywalls and membership requirements limited access to just more than half of these documents (23 of 45 [51%]); thus, they were excluded from data analysis because they were not freely available online. The included documents were mostly websites (13 of 22 [59%]) and videos (6 of 22 [27%]) offered exclusively in English (19 of 22 [86%]). Many of the documents did not report a creation year (10 of 22 [45%]), author information (10 of 22 [45%]), or funding sources (19 of 22 [86%]). Countries that developed the greatest amount of training were the US (8 of 22 [36%]), United Kingdom (4 of 22 [18%]), and Germany (3 of 22 [14%]). The main training formats were online modules (13 of 22 [59%]) and webinars (5 of 22 [23%]) and took less than 1 hour to complete (15 of 22 [68%]). Topics that were frequently included were an overview of the peer review process (18 of 22 [82%]), synthesis of a peer review report (20 of 22 [91%]), and critical appraisal of data (18 of 22 [82%]). Conversely, critical appraisal of clinical trials (4 of 22 [18%]), statistics (4 of 18 [3%]), and reporting guidelines (9 of 22 [41%]) were less commonly included.
Conclusions
This systematic review identified a comprehensive list of available online training material for scholarly peer review of biomedical journals and an analysis of their characteristics.
1Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada, jwill135@uottawa.ca; 2Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada; 3Department of Biology, Faculty of Science, University of Ottawa, Ottawa, ON, Canada; 4University of Ottawa Heart Institute, Ottawa, ON, Canada; 5School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
Conflict of Interest Disclosures
David Moher is an associate director of the Peer Review Congress but was not involved in the review or decision of this abstract.