Abstract
Assessment of Concordance Between Reports of Clinical Studies Posted as medRxiv Preprints and Corresponding Publications in Peer Reviewed Journals
Guneet Janda,1 Vishal Khetpal,2 Xiaoting Shi,3 Joseph S. Ross,4,5,6 Joshua D. Wallach7
Objective
The study characteristics, results, and interpretations described in preprints of clinical studies that are subsequently published in high-impact journals are broadly concordant.1. Given that studies published in high-impact journals may represent the highest-quality studies, it is necessary to evaluate concordance for a larger sample of clinical studies posted as preprints and subsequently published in any journal, regardless of impact factor (IF).
Design
In this cross-sectional analysis, preprints posted on medRxiv in September 2020 were identified. Four evaluators determined how many preprints were subsequently published in peer-reviewed journals as of March 2022, calculating the time from preprint posting to publication. For preprints with multiple versions, the most recent version prior to journal acceptance was selected. Preprints updated after journal acceptance were excluded. For preprint-journal article pairs describing clinical trials, observational studies, and meta-analyses that measured health-related outcomes, sample size, primary end points, corresponding results, and overarching conclusions were abstracted. Results from primary end points were considered concordant only if they contained the same information or had numerical equivalence (eg, identical effect size estimates and 95% CIs or P values from inferential analyses). Rates of concordance were compared between preprints and corresponding journal articles overall and by focus on COVID-19 and journal IF.
Results
Among 1399 preprints first posted on medRxiv in September 2020, there were 623 modeling studies (44.5%), 280 observational studies (20.0%), 90 systematic reviews or meta-analyses (6.4%), 42 clinical trials (3.0%), and 364 articles with other study designs (26.0%). As of March 2022, there were 680 preprints (48.6%) published a median (IQR) of 5 (3-7) months after preprint posting. Among 331 preprint-publication pairs describing clinical trials, observational studies, or meta-analyses, 182 pairs (55.0%) were related to COVID-19. Of 325 pairs reporting sample sizes in both sources, 290 pairs (89.2%) were concordant. Of 35 pairs with discordant sample sizes, 20 pairs (57.1%) had larger samples in the journal publication. There were 328 pairs (99.1%) with concordant and 3 pairs (0.8%) with discordant primary end points. Among 329 pairs in which results for primary end points could be compared, 290 pairs (88.1%) were concordant. Two-thirds of 39 discordant pairs (26 pairs [66.7%]) had effect size estimates in the same direction and were statistically consistent. Overall, 323 pairs (97.6%) had concordant study interpretations, including 32 of 39 pairs with discordant primary end point results (82.1%). Pairs with corresponding publications in journals with an IF of 10 or higher had lower concordance rates for sample size (17 of 24 pairs [70.8%] vs 235 of 258 pairs [91.1%]; P = .01) and results (16 of 23 pairs [70.0%] vs 235 of 261 pairs [90.0%]; P = .004).
Conclusions
Most clinical studies posted as preprints on medRxiv and subsequently published in peer-reviewed journals had concordant study characteristics, results, and interpretations, similar to what has been previously observed among preprints published in the highest-impact journals.1
Reference
1. Shi X, Ross JS, Amancharla N, Niforatos JD, Krumholz HM, Wallach JD. Assessment of concordance and discordance among clinical studies posted as preprints and subsequently published in high-impact journals. JAMA Netw Open. 2021;4(3):e212110. doi:10.1001/jamanetworkopen.2021.2110
1Yale School of Medicine, New Haven, CT, USA; 2Department of Medicine, Warren Alpert Medical School of Brown University, Providence, RI, USA; 3Department of Environmental Health Sciences, Yale School of Public Health, New Haven, CT, USA; 4Section of General Internal Medicine, Yale School of Medicine, New Haven, CT, USA; 5Yale-New Haven Hospital Center for Outcomes Research and Evaluation, New Haven, CT, USA; 6Department of Health Policy and Management, Yale School of Public Health, New Haven, CT, USA; 7Department of Environmental Health Sciences, Yale School of Public Health, New Haven, CT, USA, joshua.wallach@yale.edu
Conflict of Interest Disclosures
Guneet Janda receives research funding from the Yale-Mayo Clinic Center for Excellence in Regulatory Science and Innovation (CERSI) Scholars Award Program. Vishal Khetpal serves in an advisory role for Necessary Ventures. Xiaoting Shi is supported by the China Scholarship Council. Joseph S. Ross is a former associate editor of JAMA Internal Medicine and a current research editor at The BMJ; receives research support through Yale University from Johnson & Johnson to develop methods of clinical trial data sharing, the Medical Device Innovation Consortium as part of the National Evaluation System for Health Technology, the US Food and Drug Administration (FDA) for CERSI, the Agency for Healthcare Research and Quality, the National Heart, Lung, and Blood Institute of the National Institutes of Health (NIH), and the Laura and John Arnold Foundation to establish the Good Pharma Scorecard at Bioethics International; and is an expert witness at the request of relator’s attorneys, the Greene Law Firm, in a qui tam suit alleging violations of the False Claims Act and the Anti-Kickback Statute against Biogen. Joseph S. Ross reported cofounding medRxiv, the preprint server for health sciences research, and Joshua D. Wallach reported being a medRxiv affiliate; these individuals did not receive compensation for their roles. Joshua D. Wallach is supported by the National Institute on Alcohol Abuse and Alcoholism of the NIH.