Abstract
Assessment and Comparison of Preprints and Peer-Reviewed Publications of Reporting Characteristics of Randomized Clinical Trials of Pharmacologic Treatment for COVID-19
Philipp Kapp,1,2,3,4 Laura Esmail,1,2,3 Lina Ghosn,1,2,3 Philippe Ravaud,1,2,3 Isabelle Boutron1,2,3
Objective
Due to the pandemic, preprint servers contained up to 25% more trials, whereas some medical journals accelerated their editorial processes to ensure the rapid dissemination of findings.1 Concerns regarding quality and transparency rose.2 This meta-study assessed the transparency, completeness, and consistency of COVID-19 reports and whether there was an improvement after journal peer review.
Design
The Cochrane COVID-19 Study Register and L OVE COVID-19 platform were searched to identify all reports (preprints or peer-reviewed publications) of randomized clinical trials (RCTs) assessing pharmacologic interventions for the treatment of COVID-19, up to May 31, 2021. A standardized, online data-extraction form was developed. Data extraction covered general trial characteristics, transparency indicators (eg, trial registration and data sharing statement), completeness of reporting (eg, 10 of the most important Consolidated Standards of Reporting Trials [CONSORT] 2010 items3), and changed outcomes between report and registry (ie, switched outcomes). Two reviewers were trained and assessed 20 trials separately, with an agreement of 96.6% and a κ coefficient of 0.87. A third person resolved disagreements, when needed. A single reviewer assessed all remaining trials. For all trials published first as a preprint, a systematic search was performed for a subsequent publication in peer-reviewed journals, up to October 7, 2021. In a second step, the peer-reviewed journal publication was assessed and compared with the matched preprint.
Results
A total of 251 trial reports were identified: 121 peer-reviewed journal publications (48%) and 130 preprints (52%). Approximately half of the trials were prospectively registered (140 [56%]); 38% (n = 95) made their full protocols or statistical analysis plan available, and 29% (n = 72) provided access to their statistical analysis plan report. A data sharing statement was available in 68% of the reports (n = 170); 91% stated their willingness to share. Only 32% of the trials (n = 81) completely defined the prespecified primary outcome measures; 57% (n = 143) reported the process of allocation concealment. Overall, 51% (n = 127) adequately reported results for the primary outcomes, whereas only 14% of the trials (n = 36) adequately described harms. Primary outcome(s) reported in trial registries and published reports were inconsistent in 49% of the trials (n = 104); only 15% (n = 16) disclosed outcome switching in the report. There were no major differences between preprints and peer-reviewed publications. Of the 130 RCTs published as a preprint, 78 were then published in a peer-reviewed journal (median delay to journal publication, 94 days; IQR, 55-168 days). There was no major improvement after the peer review process.
Conclusions
Lack of transparency, completeness, and consistency of reporting are important barriers to trust, interpretation, and synthesis in COVID-19 clinical trials. Peer-reviewed publications did not report the items assessed better than preprints. The comparison of paired reports published as a preprint and as a peer-reviewed publication did not indicate major improvements.
References
1. Else H. How a torrent of COVID science changed research publishing—in 7 charts. Nature. 2020;588(7839):553. doi:10.1038/d41586-020-03564-y
2. Besançon L, Peiffer-Smadja N, Segalas C, et al. Open science saves lives: lessons from the COVID-19 pandemic. BMC Med Res Methodol. 2021;221(1):117. doi:10.1186/s12874-021-01304-y
3. Chauvin A, Ravaud P, Moher D, et al. Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study. BMC Med. 2019;17(1):205. doi:10.1186/s12916-019-1436-0
1Université de Paris, INSERM, INRAE, CNAM, Centre de Recherche en Épidémiologie et Statistiques, Paris, France, kapp@ifem.uni-freiburg.de; 2Centre d’Épidémiologie Clinique, AP-HP, Hôpital Hôtel-Dieu, Paris, France; 3Cochrane France, Paris, France; 4Institute for Evidence in Medicine (for Cochrane Germany Foundation), Medical Center University of Freiburg, Freiburg, Germany
Conflict of Interest Disclosures
Isabelle Boutron is a member of the Peer Review Congress Advisory Board but was not involved in the editorial review or decision for this abstract. No other disclosures were reported.
Funding/Support
This study was part of the COVID-NMA initiative, which received funding from Université de Paris, Assistance Publique Hôpitaux de Paris, INSERM, Cochrane France (Ministry of Health), the French Ministry of Higher Education and Research, Agence Nationale de la Recherche, and the World Health Organization.