International Congress on
Peer Review and Scientific Publication

Ayu Putu Madri Dewi,¹ Melissa L. Rethlefsen,² Sara Schroter,^3,4 Florian Naudet,^5,6 Nicholas J. DeVito,⁷ Constant Vinatier,⁵ Inge Stegeman,^8,9 Mariska Leeflang,¹ Gowri Gopalakrishna^1,10

Objective

The BMJ is one of the first journals to require authors to share analytic codes from all studies as well as data from all trials,¹ but whether this enhances reproducibility has not been tested. We will assess whether adding an open science checklist to the journal peer review process improves the reproducibility of scientific findings in submitted manuscripts to The BMJ and BMJ Open.

Design

This study is registered on Open Science Framework² and follows the SPIRIT guidelines. Reproducibility was defined as obtaining the same primary result when rerunning analyses using the original data and code. We used a 2-arm parallel randomized controlled trial design, equal allocation, and random block randomization (block sizes of 4, 6, and 8; generated in R version 4.3.2 [The R Foundation]). We hypothesized that the intervention will increase reproducibility of primary outcomes from 50% in the control arm to 70% in the intervention arm (90% power; α = .05). Our minimum sample size is 260 manuscripts (minimum 130 manuscripts per arm). Eligible manuscripts include quantitative observational or experimental designs and systematic reviews containing a meta-analysis. In the intervention arm, the research team completed an open science checklist assessing whether a set of predefined open science items³ (Table 25-1009) were reported in the manuscripts. The completed checklist was shared with the authors as part of the standard peer review process. Manuscripts in the control arm followed the usual peer review process. The primary outcome was the difference in reproducibility of the authors’ reported primary outcomes between the 2 arms, as assessed by blinded outcome assessors. The secondary outcome was the proportion of manuscripts with openly shared data and code. If the data, code, or methods were unavailable or only available on request, authors were not contacted, as they were blinded to the study, and the outcome assessors did not proceed. Randomization began on March 3, 2025, with outcome assessments conducted after the first manuscript revision.

Results

By April 18, 2025, we had completed randomization of 403 manuscripts. This includes open science intervention in 22 of 43 manuscripts from The BMJ and 179 of 360 manuscripts from BMJ Open. As of June 5, 2025, 78 manuscripts had received a decision letter, including 39 rejections, while the remainder proceeded to reproducibility assessment. Analysis is ongoing.

Conclusions

This study will provide robust evidence on whether incorporating an open science checklist into the peer review process improves the reproducibility of scientific findings.

References

1. Loder E, Macdonald H, Bloom T, Abbasi K. Mandatory data and code sharing for research published by The BMJ. BMJ. 2024;384:q324. doi:10.1136/bmj.q324

2. Dewi APM, Rethlefsen ML, Schroter S, et al. Measuring the efficacy of an intervention to improve reproducibility of scientific manuscripts: study protocol for a randomized controlled trial. OSF. Accessed July 2, 2025. https://osf.io/b5g6y/

3. Cobey KD, Haustein S, Brehaut J, et al. Community consensus on core open science practices to monitor in biomedicine. PLoS Biol. 2023;21(1):e3001949. doi:10.1371/journal.pbio.3001949

¹Epidemiology and Data Science Department, Amsterdam UMC, Amsterdam, the Netherlands; ²Health Sciences Library and Informatics Center, University of New Mexico, Albuquerque, NM, US; ³BMJ Publishing Group, London, UK; ⁴Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, London, UK; ⁵Univeristy of Rennes, CHU Rennes, INSERM, EHESP, Irset (Institut de recherche en santé, environnement et travail), UMR_S 1085, Rennes, France; ⁶Institut Universitaire de France, Paris, France; ⁷Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK, nicholas.devito@phc.ox.ac.uk; ⁸Department of Otorhinolaryngology and Head & Neck Surgery, University Medical Center Utrecht, Utrecht, the Netherlands; ⁹Brain Center, University Medical Center Utrecht, Utrecht, the Netherlands; ¹⁰Department of Epidemiology, Faculty of Health, Medicine, and Life Sciences Maastricht University, Maastricht, the Netherlands.

Conflict of Interest Disclosures

None reported.