Abstract
Feasibility of a Randomized Controlled Trial Comparing Results-Blind Peer Review vs Standard Peer Review for Reducing Publication Bias in an Open Peer Review Journal
Katherine S. Button,1 Anna Clark,2 Tim Shipley,2 Liz Bal2
Objective
To assess the feasibility of conducting a randomized controlled trial comparing the effectiveness of results-blind review with standard peer review on reducing publication bias in BMC Psychology, an open access, open peer review journal. Publication bias is defined as an excess of positive relative to null findings in the published literature, particularly among studies of low methodological quality.
Design
Single-arm feasibility study to estimate key design parameters for a full-scale trial. Parameters include proportion of positive, unclear, and null findings in the accepted articles (primary outcome) and optimizing the criteria for their classification; methodological quality of the published articles (secondary outcome) and design of measures to assess quality; willingness of authors and reviewers to participate; editorial decisions at each stage including reversal of acceptance-in-principle decisions; and time needed to collect and analyze data. We also assess acceptability of the intervention to authors, editors, and reviewers and optimize technical workflow to minimize the editorial workload required for subsequent BMC journals to participate in the trial. Results-blind review where decisions to “accept in principle” and “reject” are based on review of a partial manuscript containing the rationale and methods sections alone, with results and discussion omitted. Full versions of manuscripts “accepted in principle” are then rereviewed by the same reviewers, where the decision to publish can only be reversed if the results and discussion deviate unjustifiably from the stated aims and methods.
Results
Recruitment commenced on December 1, 2016. Of the 50 research articles submitted since, 14 (28 %) have entered into the results-blind feasibility study, 6 of which have received a first decision (2 accept in principle, 3 revise, and 1 reject and resubmit), and 2 final decisions (1 accept). The mean (SD) time to first decision of 46 (10) days is marginally lower than that of the journal average for standard submissions (82 [11]). The recruitment target for the feasibility phase is 20 research articles.
Conclusions
To date, 28% of authors are willing to participate in the study, and we are working to further improve uptake. Based on these preliminary data, editorial decisions, peer-review times, and acceptance rates are no worse and may be slightly better than the journal’s averages, providing support for the feasibility of the intervention. Work optimizing criteria for classifying publication bias outcome measures is ongoing.
1Department of Psychology, University of Bath, Bath, UK, k.s.button@bath.ac.uk; 2BioMed Central, London, UK
Conflict of Interest Disclosures:
Drs Clark, Shipley, and Bal are employees of BioMed Central. Dr Button declares no conflicts of interest.
Acknowledgments:
We thank the useful feedback on the implementation of this results-free peer review trial from the BMC Psychology Editorial Board and the Research Integrity group (http://www.biomedcentral.com/about/who-we-are/research-integrity-group), especially Maria Kowalczuk. We also thank members of the Editorial Office, who have helped with the implementation as part of the peer-review workflow for BMC Psychology, especially Ruth Baker and Sanam Sadarangani.