Abstract
Analysis of Cancer Research Discussion Text and References in High Impact Factor Journals for Possible Indicators of Paper Mills
Annie Whamond,1 Adrian G. Barnett,2 Jennifer A. Byrne1,3
Objective
Paper mills are unethical organizations that provide low-value or fraudulent content to client authors.1 Although image manipulation and incorrect reagent detection tools are available,1 these approaches can require reviewers and readers to learn new skills. Scaled manuscript production through templates may also result in paper mill articles showing other common features. We aimed to help readers identify possible indicators of paper mill support by simply scanning article text and references. We focused on discussion sections for this study.
Design
We used an exploratory, cross-sectional design2 to develop a descriptive analysis of discussion text and references in high Impact Factor (IF) cancer journals, defined as IF of 7 or higher for journal categories oncology, biochemistry and molecular biology, or cell biology. We downloaded the Retraction Watch database on October 16, 2024, and filtered for retraction reason paper mill, subject cancer, and journals on our high IF list. To identify comparison article cohorts, we randomly sampled original cancer articles in (1) the same 8 journals as retracted articles or (2) 8 independent journals with no paper mill retractions and high IF maintained for 20 years or longer. For each article, we recorded the percentages of references first cited in the introduction, methods, results, or discussion section. Focusing on discussion sections, we classified each sentence as providing background, summary, comparison, interpretation or implication, limitation, or future direction3 and recorded whether individual discussion sentences cited new references.
Results
We found 22 retracted paper mill articles from 8 journals published between June 21, 2016, and June 16, 2022. Both comparison groups (50 articles each) were restricted to articles published between January 1, 2016, and October 31, 2024. Articles in all 3 groups included similar numbers of total references per article (median [IQR]: 42 [36-46] retracted articles; 47 [36-55] articles in same journals; 51 [45-60] articles in independent journals) and percentages of discussion sentences per article (median [IQR]: 34 [30-38] retracted articles; 40 [30-48] same journals; 34 [26-43] independent journals). Analyses of discussion sections indicated that retracted paper mill articles included higher percentages of references that were first cited in the discussion and higher percentages of discussion sentences that described background information and cited new references (Figure 25-1022).
Conclusions
Our analyses suggest that some discussion sections in retracted paper mill cancer research articles reiterate background information that is supported by new and possibly superfluous references. While recognizing that genuine studies also cite new references in discussion sections, superficial and redundant second introductions in discussion sections could help readers identify potentially problematic articles in high IF cancer journals, particularly when combined with other features of paper mill support.1
References
1. Byrne JA, Abalkina A, Akinduro-Aje O, et al. A call for research to address the threat of paper mills. PLoS Biol. 2024;22(11):e3002931. doi:10.1371/journal.pbio.3002931
2. Kesmodel US. Cross-sectional studies—what are they good for? Acta Obstet Gynecol Scand. 2018;97(4):388-393. doi:10.1111/aogs.13331
3. Toronto CE, Remington R. Discussion and conclusion. In: A Step-by-Step Guide to Conducting an Integrative Review. Toronto CE, Remington R, eds. Springer International; 2020:71-84.
1School of Medical Sciences, Faculty of Medicine and Health, University of Sydney, Sydney, New South Wales, Australia, jennifer.byrne@health.nsw.gov.au; 2School of Public Health and Social Work, Queensland University of Technology, Kelvin Grove, Queensland, Australia; 3NSW Health Statewide Biobank, NSW Health Pathology, Camperdown, New South Wales, Australia.
Conflict of Interest Disclosures
None reported.
Funding/Support
Adrian G. Barnett and Jennifer A. Byrne acknowledge grant funding from the National Health and Medical Research Council of Australia (ideas grant APP2029249). This grant supports Annie Whamond’s PhD candidature.
Role of Funder/Sponsor
The funding body played no role in the study design, data collection, management, analysis, or interpretation, and will play no role in the writing of any report, or the decision to submit the report for publication.