Prevalence of Reproducible Health Sciences Research: A Systematic Review and Meta-Analysis

Niklas Bobrovitz,^1,2 Harriet Ware,² Corson Johnstone,³ Juliane Kennett,³ Stephana Julia Moss,⁴ Liam Whalen-Browne,³ Faizan Khan,⁵ Benjamin Fletcher,² Daniel J. Niven,^3,6,7 Henry T. Stelfox^3,6,8

Objective

To synthesize evidence on the prevalence and measurement of reproducibility in health sciences research.

Design

This was a systematic review with meta-analysis registered a priori.¹ We searched MEDLINE, Embase, Web of Science, the Cochrane Controlled Trials Registry and Database of Systematic Reviews, and grey literature published from January 1, 2000, to May 31, 2023. We included English-language articles in the health sciences that reported the prevalence of reproducible research or the prevalence of studies dedicated to reproducing research. We included studies published after 2000 to ensure relevance to current practice. Two authors independently screened articles in duplicate. Extractions were completed by one reviewer and verified by a second. The prevalence of reproducible research was defined as the proportion of research findings or studies for which the methods could be exactly repeated (methodological), the results could be corroborated by a study using the same or similar methods (results), or qualitatively similar conclusions could be made based on a study reproduction (inferential). Risk of bias was assessed using a modified Joanna Briggs Institute checklist. We meta-analyzed estimates using random effects models. To explore between-study heterogeneity, we conducted univariable and multivariable meta-regression using generalized linear mixed models. The level of significance was set at P < .05.

Results

A total of 177 studies published between 2001 and 2023 were included, covering all health sciences domains: biomedical/preclinical (n = 56), clinical (n = 106), health/social care services (n = 2), population health (n = 5), and multiple domains (n = 8). Studies reported on 3 types of reproducibility: methodological (n = 68), results (n = 129), and inferential (n = 5). There were 7 methodological approaches for conducting reproducibility research and 38 metrics to quantify reproducibility. The pooled estimated prevalence of reproducible research was 36.2% (95% CI, 29.7%-43.3%; I² = 99.8%; n = 180 estimates) with variation by research domain: 30.3% (95% CI, 20.3%-42.6%; n = 59 estimates) for biomedical/preclinical, 38.3% (95% CI, 30.0%-47.4%; n = 106 estimates) for clinical, 66.6% (95% CI, 20.7%-93.9%; n = 3 estimates) for health/social care services, and 36.5% (95% CI, 0.0%-100.0%; n = 3 estimates) for population health. There was a nonsignificant decline over time between 2001 and 2023 of −4.7% (95% CI −10.1 to 1.1%; P = .11) per year in the odds of reproducibility (Figure 25-1041). The pooled estimated prevalence of studies dedicated to reproducing research was 1.4% (95% CI, 0.40%-4.10%; I² = 99.2%; n = 40 estimates). Most estimates were at moderate risk of bias (n = 193), with some at high (n = 25) or low (n = 2) risk. Meta-regression of publication year, research domain, type of reproducibility, metric to quantify reproducibility, and risk of bias explained minimal estimate heterogeneity.

Conclusions

We found that an estimated one-third of health sciences research was reproducible and 1 in 100 studies were dedicated to reproducing research. Further investigation is needed to understand heterogeneity in estimates. Standardized strategies for quantifying reproducibility should be developed. Reproducibility studies are needed in health/social care services and population health research.

Reference

1. Bobrovitz N. The reproducibility of health research. Open Science Framework. 2021. https://osf.io/3fhd9

¹Department of Emergency Medicine, Cumming School of Medicine, University of Calgary, Canada, njhbobro@ucalgary.ca; ²Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Canada; ³Department of Critical Care Medicine, Cumming School of Medicine, University of Calgary, Canada; ⁴Department of Pediatrics, Division of Infectious Diseases, IWK Health Centre, Dalhousie University, Canada; ⁵Hotchkiss Brain Institute, Cumming School of Medicine, University of Calgary, Canada; ⁶O’Brien Institute for Public Health, University of Calgary, Canada; ⁷Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Canada; ⁸Faculty of Medicine and Dentistry, University of Alberta, Canada.

Conflict of Interest Disclosures

None reported.

Funding/Support

The work was funded by the Canadian Institutes of Health Research.

Role of the Funder/Sponsor

The funding body had no role in the design or conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.