International Congress on
Peer Review and Scientific Publication

Benjamin Speich,^1,2 Erika Mann,³ Christof M. Schönenberger,² Katie Mellor,¹ Alexandra N. Griessbach,² Paula Dhiman,^1,4 Pooja Gandhi,^5,6 Szimonetta Lohner,^7,8 Arnav Agarwal,^9,10 Ayodele Odutayo,^1,11 Iratxe Puebla,¹² Alejandra Clark,¹² An-Wen Chan,¹³ Michael M. Schlussel,^1,4 Philippe Ravaud,^14,15 David Moher,^16,17 Matthias Briel,^2,9 Isabelle Boutron,^14,15 Sara Schroter,¹⁸ Sally Hopewell¹

Objective

Reporting guidelines have been available since 1994. Numerous studies have shown that adherence to reporting guidelines is suboptimal,^1,2 raising the question of whether a specific targeted intervention for peer reviewers might improve reporting. The aim of this study was to evaluate whether asking peer reviewers, via email, to check if specific reporting guideline items were adequately reported in the submitted manuscripts they were reviewing would improve adherence to reporting guidelines in published articles.

Design

Two parallel-group superiority randomized controlled trials (RCT-1 and RCT-2) using submitted manuscripts as the unit of randomization. RCT-1 focused on RCT protocols and how well they were reported considering the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) guidelines, and RCT-2 focused on RCT results publications and the reporting of CONSORT (Consolidated Standards of Reporting Trials) items. Manuscripts in both RCTs were randomized (1:1) to intervention or control; the control group received usual journal practice. RCT-1 included manuscripts containing RCT protocols submitted from June 2020 to May 2021 to BMJ Open that were sent for peer review (https://osf.io/z2hm9). The RCT-2 trial³ included manuscripts describing RCT primary results, submitted from July 2019 to July 2021 to 1 of 7 journals (5 BMJ Publishing Group; 2 Public Library of Science [PLOS]). In the intervention group (both trials), peer reviewers received an email from the journal reminding them to check if items were adequately reported in the manuscript. In RCT-1, these were the 10 most important and poorly reported SPIRIT items and for RCT-2, the 10 most important and poorly reported CONSORT items. In both RCTs, peer reviewers and authors were not informed of the purpose of the study and outcome assessors were blinded. The primary outcome was the difference in the mean proportion of adequately reported 10 SPIRIT and 10 CONSORT items between intervention and control in the final published article.

Results

In RCT-1, 245 manuscripts were randomized. Of those, 178 were published (90 intervention; 88 control). A mean proportion of 46.1% (95% CI, 41.8%-50.4%) of the 10 SPIRIT items were adequately reported in the intervention group and 45.6% (95% CI, 41.7%-49.4%) in the control group (mean difference, 0.5%; 95% CI, −5.2% to 6.3%) (Figure 10). In RCT-2, of the 511 randomized manuscripts, 243 were published (121 intervention; 122 control). A total of 67.4% (95% CI, 63.8%-71.1%) of the 10 CONSORT items were adequately reported in the intervention group and 65.9% (95% CI, 61.9%-69.9%) in the control group (mean difference, 1.5%; 95% CI, −3.8% to 6.9%) (Figure 10).

Conclusions

Journals asking peer reviewers, via email, to check if the most important and poorly reported items are adequately reported in submitted manuscripts did not improve the reporting completeness of the final published article.

References

1. Samaan Z, Mbuagbaw L, Kosa D, et al. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169-188.

2. Jin Y, Sanger N, Shams I, et al. Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? a systematic review of reviews: an update. J Multidiscip Healthc. 2018;11:495-510.

3. Speich B, Schroter S, Briel M, et al. Impact of a short version of the CONSORT checklist for peer reviewers to improve the reporting of randomised controlled trials published in biomedical journals: study protocol for a randomised controlled trial. BMJ Open. 2020;10(3):e035114.

¹Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK; ²Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland, benjamin.speich@usb.ch; ³PLOS, Public Library of Science, San Francisco, CA, USA; ⁴The EQUATOR Network, Oxford, UK; ⁵Rehabilitation Sciences Institute, University of Toronto, Toronto, ON, Canada; ⁶Swallowing Rehabilitation Research Laboratory, Toronto Rehabilitation Institute–University Health Network, Toronto, ON, Canada; ⁷Cochrane Hungary, Clinical Centre of the University of Pécs, Medical School, University of Pécs, Pécs, Hungary; ⁸Department of Public Health Medicine, Medical School, University of Pécs, Pécs, Hungary; ⁹Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada; ¹⁰Division of General Internal Medicine, Department of Medicine, McMaster University, Hamilton, ON, Canada; ¹¹Applied Health Research Centre, Li Ka Shing Knowledge Institute of St Michael’s Hospital, Toronto, ON, Canada; ¹²PLOS ONE, Public Library of Science, San Francisco, CA, USA, and Cambridge, UK; ¹³Department of Medicine, Women’s College Research Institute, Women’s College Hospital, University of Toronto, Toronto, ON, Canada; ¹⁴Centre d’Épidémiologie Clinique, Hôpital Hôtel-Dieu, Assistance Publique Hôpitaux de Paris (APHP) Paris, France; ¹⁵Université de Paris, CRESS, INSERM, INRA, Paris, France; ¹⁶Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada; ¹⁷School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada; ¹⁸The BMJ, London, UK

Conflict of Interest Disclosures

Erika Mann and Alejandra Clark are employed by the Public Library of Science. Paula Dhiman, Michael M. Schlussel, Philippe Ravaud, and David Moher are members of the EQUATOR (Enhancing the Quality and Transparency of Research) network. During the design and initial implementation, Iratxe Puebla was an employee by the Public Library of Science. An-Wen Chan and David Moher are authors of the SPIRIT 2013 Statement (Standard Protocol Items: Recommendations for Interventional Trials). David Moher, Sally Hopewell, and Isabelle Boutron are members of the Consolidated Standards for Reporting Trials (CONSORT) group and authors of the CONSORT 2010 statement. Sara Schroter is employed by The BMJ. David Moher is an associate director and An-Wen Chan and Isabelle Boutron are advisory board members of the International Congress on Peer Review and Scientific Publication but were not involved in the review or decision for this abstract. No other disclosures were reported.

Funding/Support This study was supported in part by the Swiss National Science Foundation.

Additional Information The study was registered on the Open Science Framework (https://osf.io/c4hn8).