Quotation Inaccuracy in Medicine: A Systematic Review and Meta-Analysis
Abstract
Christopher Baethge,1,2 Hannah Jergas3
Objective
Although quotations are central to scientific publications, they often misrepresent the cited reference.1 Inaccurate quotations may mislead readers, undermine the line of argument of a paper, and threaten the integrity of the scientific record.2,3 We updated an earlier meta-analysis1 to assess improvement in recent years.
Design
This systematic review and meta-analysis, preregistered on Open Science Framework (OSF), was conducted following the Cochrane Handbook and PRISMA statement. It focuses on quotation inaccuracy—errors of content—as opposed to bibliographical errors, often referred to as citation errors. We searched MEDLINE, PubMed Central via PubMed, and Web of Science from January 1, 2014, through October 1, 2023, for studies on quotation errors in the medical literature, with no date, language, or source restrictions, and we hand-searched all reference lists of included studies. Literature screening, data extraction, and risk of bias estimation were carried out by 2 authors independently. We adopted the original studies’ definitions1 of major inaccuracies (ie, not at all in accordance with the claim of the original authors), minor inaccuracies (ie, inconsistencies and factual errors not severe enough to contradict a statement by citing authors), secondary quotations (ie, quotations of sources referencing an original study), and total quotation errors. Summary proportions of quotation errors were estimated in random-effects meta-analyses (log-transformed proportions and DL τ2 estimates), with heterogeneity measures, 95% confidence intervals, and 95% prediction intervals. Time trends were investigated in meta-regressions.
Results
Based on 46 studies included, with 32,074 quotations/references checked, 16.9% were incorrect (95% CI, 14.1%-20.0%). Of these, 8.0% (95% CI, 6.4%-10.0%) were major inaccuracies and 7.8% (95% CI, 5.7%-10.5%) were minor inaccuracies. Heterogeneity throughout all meta-analyses was high (Table 24-0811). Rates of total errors have not improved during the last decade (slope: −0.002; 95% CI, −0.03 to 0.02; P = .85) and neither have rates of major errors (slope: 0.002; 95% CI, −0.02 to 0.03; P = .89). There were no associations with risk of bias, publication bias, number of references, or medical specialty (surgical, nonsurgical, other), but the total error rate was statistically significantly (P = .04; binomial, n = 25) and negatively correlated with Journal Impact Factor (−0.25; Spearman, n = 23). Interrater reliability was sufficient (Cohen κ: 0.73), and summary estimates were supported by sensitivity analyses employing arcsine square root data transformation and Sidik and Jonkman τ2 estimates.
Conclusions
Quotation inaccuracy remains prevalent in scientific medical texts. Despite seemingly broad awareness among journal editors and probably among many authors, there is no evidence of improvement in recent years. It is uncertain how the utilization of artificial intelligence in writing articles and in double-checking quotation inaccuracy will impact the problem, but human effort will likely remain important in creating and vetting quotes—on the part of authors, editors, reviewers, and readers.
References
1. Jergas H, Baethge C. Quotation accuracy in medical journal articles—a systematic review and meta-analysis. PeerJ. Published online October 27, 2015. doi:10.7717/peerj.1364
2. Baethge C. Importance, errors, and patterns of quotations to psychiatric original articles. Pharmacopsychiatry. 2020;53(6):247-255. doi:10.1055/a-1167-35673
3. Peoples N, Østbye T, Yan LL. Burden of proof: combating inaccurate citation in biomedical literature. BMJ. Published online November 6, 2023. doi:10.1136/bmj-2023-076441
1Deutsches Ärzteblatt and Deutsches Ärzteblatt International, Editorial Offices, Cologne, Germany, baethge@aerzteblatt.de; 2Department of Psychiatry and Psychotherapy, Faculty of Medicine, University of Cologne, Cologne, Germany; 3Department of Neurology, Faculty of Medicine, University of Cologne, Cologne, Germany.
Conflict of Interest Disclosures
Both authors have published on this topic before. Christopher Baethge is employed by a general medical journal (Deutsches Arzteblatt, Deutsches Arzteblatt International).
Acknowledgment
The authors gratefully acknowledge Joan Albert Hammerstein’s help in screening the literature.
Additional Information
This study is registered on Open Science Framework (OSF) at https://osf.io/95rej/?view_only=8c5d2b51a8814278bff1fec1e40d0bb9