Abstract
Reminding Peer Reviewers to Comment on Reporting Items as Instructed by the Journal:
An Analysis of 2 Randomized Trials
Hillary Wnfried Ramirez,1,2,3 Malena Chiaborelli,1,2,3 Christof M. Schönenberger,1 Katie Mellor,4,5 Alexandra N. Griessbach,1 Paula Dhiman,4,6 Pooja Gandhi,7 Szimonetta Lohner,8,9 Arnav Agarwal,10,11 Ayodele Odutayo,5,12 Michael M. Schlussel,4,6 Philippe Ravaud,13,14 David Moher,15,16 Matthias Briel,1,10 Isabelle Boutron,13,14 Sally Hopewell,4 Sara Schroter,17,18 Benjamin Speich1,4
Objective
Two randomized controlled trials (RCTs) conducted at journal level have shown that reminding peer reviewers about the 10 most important and underreported reporting items did not improve the reporting quality in published articles.1 With this pooled in-depth analysis of peer reviewer reports, we aimed to assess at what stage the intervention failed.
Design
A subsample of peer reviewer reports from the control group (receiving no reminder) and the intervention group (receiving a reminder of the 10 most important reporting items) were analyzed. In brief, 2 blinded authors independently extracted from peer reviewer reports how many of the 10 key reporting items were flagged by peer reviewers for clarification. The main outcome of this analysis was the mean proportion of the 10 selected reporting items for which at least 1 peer reviewer requested clarification, assessed at the manuscript level. Furthermore, we assessed how many requested changes were later adequately reported in published articles.
Results
Across the RCTs, we had access to peer reviewer reports for 533 manuscripts (265 in the intervention group, assessing comments from 740 peer reviewers; and 268 in the control group, assessing comments from 719 peer reviewers). Our results indicate that reviewers in the intervention group requested clarification on more reporting items than those in the control group. Overall, reviewers in the intervention group flagged 21.1% of the 10 reporting items for clarification compared with 13.1% in the control group (mean difference, 8.0 percentage points (pp); 95% CI, 4.9-11.1 pp). However, the overall mean difference between groups was diluted from 8.0 to 4.2 pp when only assessing accepted and published articles and decreased even further to 2.6 pp when only considering changes that were then implemented by authors of manuscripts (Table 25-0956). Approximately 55% of reporting items that were criticized by peer reviewers were later adequately reported in the published article (intervention group, 173 of 308 [56.2%]; control group, 105 of 197 [53.3%]).
Conclusions
Reminding peer reviewers to check reporting items increased their focus on reporting guidelines, leading to more reporting-related requests in their review reports. However, the effect was diluted during the peer review process (particularly due to rejected articles and requests not being implemented by authors). Journals should therefore make sure that requested clarifications are adequately addressed in revised manuscripts.
Reference
1. Speich B, Mann E, Schönenberger CM, et al. Reminding peer reviewers of reporting guideline items to improve completeness in published articles: primary results of 2 randomized trials. JAMA Netw Open. 2023;6(6):e2317651.
1CLEAR Methods Center, Division of Clinical Epidemiology, Department Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland, benjamin.speich@usb.ch; 2Swiss Tropical and Public Health Institute, Basel, Switzerland; 3University of Basel, Basel, Switzerland; 4Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK; 5Clinical Outcomes Assessment, Clarivate, London, UK; 6The EQUATOR Network, Oxford, UK; 7Department of Communication Sciences and Disorders, Faculty of Rehabilitation Medicine, University of Alberta, Edmonton, AL, Canada; 8Cochrane Hungary, Medical School, University of Pécs, Pécs, Hungary; 9MTA–PTE Lendület “Momentum” Evidence in Medicine Research Group, Department of Public Health Medicine, Medical School, University of Pécs, Pécs, Hungary; 10Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada; 11Division of General Internal Medicine, Department of Medicine, McMaster University, Hamilton, ON, Canada; 12Division of Nephrology, Toronto General Hospital, University Health Network, Toronto, ON, Canada; 13Centre d’Épidémiologie Clinique, Hôpital Hôtel-Dieu, Assistance Publique Hôpitaux de Paris, Paris, France;14Université de Paris, CRESS, Inserm, INRA, Paris, France; 15Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; 16Faculty of Medicine, School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada; 17The BMJ, London, UK; 18Faculty of Public Health & Policy, London School of Hygiene & Tropical Medicine, London, UK.
Conflict of Interest Disclosures
Benjamin Speich and Matthias Briel reported receiving unrestricted grants from Moderna for studies unrelated to the presented work. Sara Schroter is employed by BMJ Publishing Group. Katie Mellor is employed by Clarivate. David Moher, Sally Hopewell, and Isabelle Boutron are members of the Consolidated Standards for Reporting Trials (CONSORT) executive board and authors of the CONSORT 2010 statement. David Moher is an author of the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) 2013 statement. David Moher, Michael M. Schlussel, Paula Dhiman, and Philippe Ravaud are members of the Enhancing the Quality and Transparency of Research (EQUATOR) network. Isabelle Boutron and David Moher are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract. No other disclosures were reported.
Funding/Support
Benjamin Speich was supported by a Return Postdoc.Mobility (P4P4PM_194496) grant from the Swiss National Science Foundation. Christof M. Schönenberger was funded by the Janggen Pöhn Foundation and the Swiss National Science Foundation (MD-PhD grant No. 323530_221860). Szimonetta Lohner was supported by the Hungarian Academy of Sciences (MTA) within the framework of the Lendület Programme.
Role of the Funder/Sponsor
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Additional Information
Both trials (CONSORT-PR and SPIRIT-PR) were prospectively registered on Open Science Framework (https://osf.io/c4hn8 and https://osf.io/z2hm9).