Abstract
Peer Reviewed Evaluation of Registered End Points of Randomized Trials (the PRE-REPORT Study)
Christopher W. Jones,1 Amanda C. Adams,2 Benjamin S. Misemer,3 Mark A. Weaver,4 Sara Schroter,5 Hayat Khan,1 Benyamin Margolis,6 David L. Schriger,7 Timothy F. Platts-Mills8
Objective
To test whether providing information from relevant clinical trial registry entries to peer reviewers evaluating trial manuscripts decreases discrepancies between registered and published trial outcomes.1,2
Design
This stepped-wedge, cluster-randomized trial studied clusters comprised of eligible clinical trial manuscripts submitted to each of 13 high-impact medical journals across a range of medical specialties between November 1, 2018, and October 31, 2019.3 All participating journals routinely published results from clinical trials. Each journal began the study in the control phase; journals were crossed over into the intervention phase in random order. Manuscripts submitted during the study period were included if they presented results from the primary analysis of a clinical trial and underwent external peer review. Peer reviewers were not told that a study was taking place. Preexisting peer review practices were unchanged during the control phase. Peer reviewers for intervention-phase manuscripts received a data sheet describing whether the trial was registered, the initial registration and enrollment dates, and the registered primary outcome(s) when enrollment began. Decision editors had access to the registry data sheets. The primary end point was the presence of a published primary outcome consistent with a prospectively defined primary outcome in the study’s trial registry (ie, registered before enrollment began), as determined independently by 2 outcome assessors blinded to each manuscript’s study arm. Linear mixed models were used to estimate outcome differences between intervention- and control-condition manuscripts. For the primary end point, use of a 1-sided test at the 5% level was prespecified, with corresponding 90% CIs, based on the assumption that the intervention was unlikely to increase outcome inconsistencies.
Results
The study included 419 submitted manuscripts. Participating journals published 105 of 243 control-phase manuscripts (43%) and 68 of 176 intervention-phase manuscripts (39%) (model-estimated difference between intervention and control, −10%; 95% CI, −25% to 4%). Among the 173 accepted manuscripts, published primary outcomes were consistent with clearly defined, prospectively registered primary outcomes in 40 of 105 control-phase manuscripts (38%) and 27 of 68 intervention-phase manuscripts (40%). There was no statistically significant difference between intervention and control phases in the primary end point (estimated difference, −6%; 90% CI, −27% to 15%; 1-sided P = .68). Prospectively registered trials were more likely to be published (117 of 251 trials [47%]) than unregistered trials (7 of 33 trials [21%]) (model-estimated difference, 29%; 95% CI, 10%-47%), but no significant difference was observed between prospectively and retrospectively registered trials (49 of 135 trials [36%] accepted; model-estimated difference, 9%; 95% CI, 1%-20%).
Conclusions
The results do not support provision of a data sheet with clinical trial registration details during peer review to increase agreement between prospectively registered and published trial outcomes. The high prevalence of retrospective registration and discrepancies between registered and published trial outcomes necessitates identification of effective interventions for these problems.
References
1. Trinquart L, Dunn AG, Bourgeois FT. Registration of published randomized trials: a systematic review and meta-analysis. BMC Med. 2018;16(1):173. doi:10.1186/s12916-018-1168-6
2. Wager E, Williams P, “Hardly worth the effort”? medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: quantitative and qualitative study. BMJ. 2013;347:f5248. doi:10.1136/bmj.f5248
3. Ellenberg SS. The stepped-wedge clinical trial: evaluation by rolling deployment. JAMA. 2018;319(6):607-608. doi:10.1001/jama.2017.21993
1Department of Emergency Medicine, Cooper Medical School of Rowan University, Camden, NJ, USA, jones-christopher@cooperhealth.edu; 2Medical Library, Cooper Medical School of Rowan University, Camden, NJ, USA; 3Department of Emergency Medicine, University of Michigan, Ann Arbor, MI, USA; 4Department of Mathematics and Statistics, Elon University, Elon, NC, USA; 5The BMJ, BMA House, London, UK; 6Rollins School of Public Health, Emory University, Atlanta, GA, USA; 7Department of Emergency Medicine, University of California Los Angeles, Los Angeles, CA, USA; 8Ophirex, Inc, Corte Madera, CA, USA
Conflict of Interest Disclosures
Christopher W. Jones reported receiving grants from AstraZeneca, Abbott, Vapotherm, and Ophirex outside the submitted work. Sara Schroter is a full-time employee at The BMJ. Benyamin Margolis is an employee of the Department of Health and Human Services in the Office of Research Integrity. David L. Schriger is an associate editor at JAMA and a deputy editor at Annals of Emergency Medicine. Timothy F. Platt-Mills is an employee of Ophirex.
Funding/Support
This work was supported by the US Department of Health and Human Services Office of Research Integrity (grant ORIIR180039).
Role of the Funder/Sponsor
The sponsor had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the abstract for presentation.
Acknowledgment
We thank the editors and staff members of the following journals for their participation in this study: Academic Emergency Medicine, American Journal of Transplantation, Annals of Emergency Medicine, Archives of Physical Medicine and Rehabilitation, British Journal of Ophthalmology, Clinical Orthopaedics and Related Research, Gastroenterology, Heart, International Journal of Cancer, Journal of the American College of Surgeons, Neurology, Surgery, and Thorax.
Additional Information
The contents of this manuscript are those of the authors and do not represent the official views of, nor an endorsement by, the Office of the Assistant Secretary of Health, the Department of Health and Human Services, or the US government. The study was registered at ISRCTN41225307.