Abstract

Reproducible Research Practices in Systematic Reviews of Therapeutic Interventions: A Cross-sectional Study

Matthew J. Page,1 Douglas G. Altman,2 Larissa Shamseer,3,4 Joanne E. McKenzie,1 Nadera Ahmadzai,5 Dianna Wolfe,5 Fatemeh Yazdi,5 Ferrán Catalá-López,5,6 Andrea C. Tricco,7,8 David Moher3,4

Objective

Biomedical researchers are increasingly encouraged to use reproducible research practices, which allow others to recreate the findings, given the original data. Such practices include providing a detailed description of the data collected and used for analysis, clearly reporting the analysis methods and results, and sharing the data set and statistical code used to perform analyses (within the journal article, a supplementary file, or a data repository). To our knowledge, there has been no investigation into how often such practices are used in systematic reviews (SRs) across different specialties. We aimed to investigate reproducible research practices used in a cross-section of SRs of therapeutic interventions.

Design

We selected articles from a database of SRs we assembled previously, which included a random sample of 300 SRs that were indexed in MEDLINE during February 2014. In the current study, we included only those SRs that focused on a treatment or prevention question and reported at least 1 meta-analysis. One author collected data on 28 prespecified items that characterized reproducible research practices from the SR article and any supplementary files; a 20% random sample was collected in duplicate. We did not contact authors of the SRs for additional information. We calculated risk ratios to explore whether reproducible research practices differed between Cochrane and non-Cochrane SRs.

Results

We evaluated 110 SRs; 78 (70.9%) were non-Cochrane articles, and 55 (50.0%) investigated a pharmacological intervention. The SRs presented a median (interquartile range) of 13 (5-27) meta-analyses. Authors of SRs reported the data needed to recreate all meta-analytic effect estimates in the SR, including subgroup meta-analytic effects and sensitivity analyses, in only 72 of 110 SRs (65.5%). This percentage was higher in Cochrane than in non-Cochrane SRs (94% vs 54%) (Figure). Despite being recommended by PRISMA, summary statistics (eg, means and SDs) of each individual study were not reported for 31 of 110 index (ie, primary or first-reported) meta-analyses (28.2%). Authors of SRs who reported imputing, algebraically manipulating, or obtaining some data from the included studies’ authors/sponsors infrequently stated which specific data were handled in this way. Only 33 SRs (30.0%) mentioned access to data sets and statistical code used to perform analyses.

Figure. Risk Ratio (RR) Associations Between Type of Systematic Review (Cochrane vs Non-Cochrane) and Reproducble Research Practices

Conclusions

Reproducible research practices in SRs of therapeutic interventions are suboptimal. Authors of SRs should make greater use of public data repositories (eg, the Systematic Review Data Repository or Open Science Framework) to share SR data sets and statistical analysis code so that others can recreate the findings, check for errors, or perform secondary analyses.

1School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia, matthew.page@monash.edu; 2UK EQUATOR Centre, Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK; 3Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; 4School of Epidemiology, Public Health, and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada; 5Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; 6Department of Medicine, University of Valencia/INCLIVA Health Research Institute and CIBERSAM, Valencia, Spain; 7Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael’s Hospital, Toronto, Ontario, Canada; 8Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada

Conflict of Interest Disclosures:

Matthew Page and Joanne McKenzie are affiliates of Cochrane Australia. Matthew Page is a coconvenor of the Cochrane Bias Methods Group. Joanne McKenzie is a coconvenor of the Cochrane Statistical Methods Group. Andrea Tricco is an author of 2 of the systematic reviews included in this study but was not involved in eligibility assessment or data collection. Drs Altman and Moher are members of the Advisory Board for the Peer Review Congress but were not involved in the review or decision for this abstract.

Funding/Support:

Matthew Page is supported by grant 1088535 from the Australian National Health and Medical Research Council Early Career Fellowship. Douglas Altman is a National Institute for Health Research Senior Investigator. Joanne McKenzie is supported by grant 1072366 from the Australian National Health and Medical Research Council Australian Public Health Fellowship. Ferrán Catalá-López is supported by the Generalitat Valenciana (PROMETEOII/2015/021). Andrea Tricco is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis. David Moher is supported in part by a University Research Chair, University of Ottawa.

Video