Abstract
Notifying Authors That They Have Cited a Retracted Article and Future Citations of Retracted Articles: The RetractoBot Randomized Controlled Trial
Nicholas J. DeVito,1 Christine Cunningham,1 Ben Goldacre1
Objective
Continued citation of retracted articles is an ongoing issue.1 RetractoBot examines whether informing authors that they cited a retracted paper reduces future citations to those articles.2
Design
This was a parallel-group, 2-group, superiority trial. We identified and randomized retracted articles in PubMed published since 2000. The study was unblinded, but primary outcome ascertainment was objective, and randomization and allocation were fully and simultaneously computer automated. Authors with publicly available contact information were notified about citations of retracted articles in the intervention group; no notifications were sent about articles in the control group. The email included a voluntary survey question on knowledge of the retractions and invited additional voluntary feedback. Retraction, contact, and citation data were obtained from PubMed and Scopus. Follow-up started 3 months after the final intervention email was sent to allow in-press manuscripts that could not be affected by our intervention a chance to publish. The primary outcome was the difference in the rate of new citations of retracted articles between groups. We used a negative binomial model adjusted for years since retracted article publication, and our stratification variable, with outliers removed. Data analysis was performed in Stata version 16. We also report the results of a survey question on knowledge of retracted citations, and a sentiment analysis on email feedback. Additional methods details, including power calculations, are available in our prespecified protocol3 and at retracted.net.
Results
We randomized 7958 retracted articles to the intervention group and 7963 to the control group (Figure 25-0938). Overall, 246,749 intervention emails were deliverable to authors between January 8 and February 7, 2024, with follow-up extending from May 8, 2024, to May 7, 2025. After removing outliers, there were 7939 citations of articles in the intervention group and 7943 in the control group. The effect of our intervention was not significant (mean citation rate, −0.007; 95% CI, −0.055 to 0.041) and held for all prespecified sensitivity analyses. We received 15,667 survey responses from notified authors with 80.6% (12,631) indicating they were unaware of all retracted citations we informed them about. We received 482 substantive pieces of email feedback, of which 57.7% (278) indicated a positive sentiment about the project (vs neutral, negative, mixed, or other).
Conclusions
Our intervention did not significantly decrease citations of retracted articles at 1 year compared with the control group. We plan future secondary analyses to determine whether an effect emerges with increased follow-up. Of the authors who responded, most were unaware of the notified retractions and had a positive view of the intervention, indicating the potential for future interventions in this area. The continued uptake of retraction notifications in citation management software, using sources such as the RetractionWatch database, may already be having an effect in this space.
References
1. Kühberger A, Streit D, Scherndl T. Self-correction in science: the effect of retraction on the frequency of citations. PLoS One. 2022;17(12):e0277814.
2. Xu H, Ding Y, Zhang C, Tan BCY. Too official to be effective: an empirical examination of unofficial information channel and continued use of retracted articles. Res Policy. 2023;52(7):104815.
3. Cunningham C, DeVito N, Goldacre B. RetractoBot—revised stage 1 registered report protocol. Published online October 31, 2023. doi:10.6084/M9.FIGSHARE.24468391.V1
1Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK, nicholas.devito@phc.ox.ac.uk.
Conflict of Interest Disclosures
Nicholas J. DeVito reported receiving funding from the Naji Foundation, the Fetzer-Franklin Memorial Fund, the European Commission, and UK Research and Innovation (UKRI) outside the submitted work. Ben Goldacre reported receiving research funding from the Bennett Foundation, the Laura and John Arnold Foundation, the National Health Service (NHS) National Institute for Health Research (NIHR), the NIHR School of Primary Care Research, NHS England, the NIHR Oxford Biomedical Research Centre, the Mohn-Westlake Foundation, NIHR Applied Research Collaboration Oxford and Thames Valley, the Wellcome Trust, the Good Thinking Society, Health Data Research UK, the Health Foundation, the World Health Organization, UKRI Medical Research Council, Asthma UK, the British Lung Foundation, and the Longitudinal Health and Wellbeing strand of the National Core Studies program outside the submitted work; he has previously been a non-executive director at NHS Digital; he also receives personal income from speaking and writing for lay audiences on the misuse of science.
Funding/Support
The RetractoBot project was funded by a grant from the Laura and John Arnold Foundation and a grant and in-kind support from Elsevier.
Role of Funder/Sponsor
The funders had no role in the study design, data collection (outside of in-kind access to elements of the Scopus API from Elsevier), data analysis, interpretation, decision to submit, or any other facet of the planning, conduct, or dissemination of the presented work.
Acknowledgment
We would like to acknowledge the full RetractoBot study team including Karolina Wartolowska, Henry Drysdale, Benjamin Feakins, Helen J. Curtis, Francis Irving, Anna Powell-Smith, and Seb Bacon.