Characteristics of Studies of Research Reproducibility in Economics, Education, Psychology, Health Sciences, and Biomedicine:
A Scoping Review
Kelly D. Cobey,1,2 Christophe A. Fehlmann,2,3 Marina Christ Franco,4,5 Ana Patricia Ayala,6 Lindsey Sikora,7 Danielle B. Rice,4,8 Chenchen Xu,4,9 John P. A. Ioannidis,10 Manoj M. Lalu,4,11,12 Alixe Ménard,4 Andrew Neitzel,4,9 Bea Nguyen,4,9 Nino Tsertsvadze,4 David Moher2,4
Reproducibility is a central tenet of research. Explicit reproducibility checks are made across different disciplines trying to assess the replicability of previously published studies. This scoping review aimed to synthesize the literature on reproducibility; describe its epidemiological characteristics, including how reproducibility is defined and assessed; and determine and compare estimates for reproducibility across different fields.
All English-language quantitative replication studies within the fields of economics, education, psychology, health sciences, and biomedicine published in 2018 or 2019 were included, as were studies that were explicitly self-described as a replication or a reproducibility study in which a previously published quantitative study is referred to and conducted again. Conference proceedings, commentaries, narrative reviews, systematic reviews, and clinical case studies were excluded. MEDLINE, Embase, PsycINFO, Cumulative Index of Nursing and Allied Health Literature (CINAHL), Education Source via EBSCOHost, ERIC, EconPapers, International Bibliography of the Social Sciences, and EconLit were searched. Retrieved documents were screened in duplicate against our inclusion criteria. The year of publication, number of authors, country of affiliation of the corresponding author, and funding were extracted. For individual replication studies, whether a registered protocol was used, whether there was contact between the replicating team and the original authors, what study design was used, and what the primary outcome was were extracted from each replication study. Finally, how replication was defined by the authors and whether the assessed study or studies successfully replicated a previous study based on this definition were also extracted. Extraction was done by 2 reviewers.
The search identified 11,224 unique documents, of which 47 were included (Figure 14). Most studies were related to either psychology (48.6%) or health sciences (23.7%). Among these 47 documents, 36 described a single replication study, while the remaining 11 reported at least 2 replications in the same paper. Less than half of the studies referred to a registered protocol. There was variability in the definitions of replication success, with studies related to psychology and health sciences tending to use a comparison of effect sizes to define replication success. Based on the definition used by the authors of each study, 95 of 177 studies (53.7%) achieved replication success.
This study gives an overview of research across 5 disciplines that explicitly set out to replicate previous research. While estimates of reproducibility vary across fields in this modest sample, so too do norms in definitions used to define reproducibility.
1University of Ottawa Heart Institute, Ottawa, ON, Canada, firstname.lastname@example.org; 2School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada; 3Department of Anaesthesiology, Clinical Pharmacology, Intensive Care and Emergency Medicine, Geneva University Hospitals, Geneva, Switzerland; 4Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada; 5School of Dentistry, Federal University of Pelotas, Pelotas, Brazil; 6Gerstein Science Information Centre, University of Toronto, Toronto, ON, Canada; 7 Health Sciences Library, University of Ottawa, Ottawa, ON, Canada; 8 Department of Psychology, McGill University, Montreal, QC, Canada; 9 Department of Medicine, University of Ottawa, Ottawa, ON, Canada; 10Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA; 11Department of Anesthesiology and Pain Medicine, The Ottawa Hospital, University of Ottawa, Ottawa, ON, Canada; 12Regenerative Medicine Program, The Ottawa Hospital, Ottawa, ON, Canada
Conflict of Interest Disclosures
David Moher is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision of this abstract. No other disclosures were reported.
Alixe Ménard is a patient author.