Effectiveness of Preregistration in Psychology
Abstract
Olmo R. van den Akker,1,2 Marcel A. L. M. van Assen,1 Marjan Bakker,1 Jelte M. Wicherts1
Objective
Preregistration,1 the practice in which researchers publish their hypotheses, study design, and/or analysis plan before collecting or analyzing their data, has become more and more established in the scientific ecosystem even though the evidence that it prevents bias has been lacking. This set of studies was conducted to assess the effectiveness of preregistration to reduce bias in psychology.
Design
In the first study, conducted from 2021 to 2024, we assessed the consistency of preregistrations and their corresponding publications (published in 40 psychology journals). These preregistered publications were published and awarded a preregistration badge by a journal or a preregistration challenge prize by the Center for Open Science from 2017 to 2020. We also assessed whether any inconsistencies were disclosed by the authors. In the second study, we compared a subset of preregistration-publication pairs for which a preregistered statistical result could be extracted with a set of nonpreregistered studies (from 37 journals) that were selected for comparability via the Web of Science related records function. Using multilevel regressions, we examined whether preregistration was associated with P hacking, operationalized as a lower proportion of statistical results and smaller effect sizes. We also looked at associations with other research characteristics, such as the number of citations, sample sizes, and the frequency of statistical errors (assessed via Statcheck2).
Results
The sample included 300 preregistration-publication pairs as well as a subset of 193 preregistration-publications pairs matched with 193 nonpreregistered studies. We found inconsistencies for 6 important study elements (operationalizations of measured, manipulated, and dependent variables, data collection procedures, statistical model, and inference criteria [Table 25-0947]). The comparison between preregistered and nonpreregistered studies did not indicate an association between preregistration and statistical significance (β = 0.01; 99% CI, −0.56 to 0.59; P = .96), effect sizes (β = −0.04; 99% CI, −0.12 to 0.04; P = .18), and statistical errors (β = −1.19; 95% CI, −2.51 to 0.13; P = .08). However, preregistered studies typically had higher sample sizes (959.6 vs 536.6; β = 0.45; 99% CI, 0.14 to 0.76; P < .001) and more citations (18.3 vs 15.1; β = 0.20; 95% CI, 0.01 to 0.40; P = .04).
Conclusions
Ideally, publications would be concordant with their preregistrations and any deviations would be reported and explained. In these studies, this ideal was often not met. Preregistration was associated with higher statistical power and more subsequent citations, but it did not appear to prevent P hacking. The studies were limited by their focus on psychology, and generalizations to other fields should therefore be avoided (although economics and political science had comparable results3).
References
1. Rice DB, Moher D. Curtailing the use of preregistration: a misused term. Perspect Psychol Sci. 2019;14(6):1105-1108. doi:10.1177/1745691619858427
2. Nuijten MB, Hartgerink CH, Van Assen MA, Epskamp S, Wicherts JM. The prevalence of statistical reporting errors in psychology (1985-2013). Behav Res Methods. 2016;48:1205-1226. doi:10.3758/s13428-015-0664-2
3. Ofosu GK, Posner DN. Pre-analysis plans: an early stocktaking. Perspect Polit. 2023;21(1):174-190. doi:10.1017/S1537592721000931
1Tilburg University, Tilburg, The Netherlands, ovdakker@gmail.com; 2QUEST Center for Responsible Research, Berlin Institute of Health at Charité, Berlin, Germany.
Conflict of Interest Disclosures
None reported.
Funding/Support
The work in this presentation was supported by a Consolidator Grant (IMPROVE) from the European Research Council (grant number 726361).
Additional Information
Both studies presented in this abstract were preregistered: the assessment of preregistration effectiveness at https://osf.io/qbhsv, and the comparison of preregistered with nonpreregistered studies at https://osf.io/7w5g2.