Annapoorani Muthiah,1 Loch Kith Lee,1 John Koh,1 Ashley Liu,1 Aidan C. Tan2
Over the past decade, there has been an exponential increase in systematic reviews and meta-analyses (SR/MAs) published in the medical literature, and some concerns have arisen that many of these are duplicate studies with redundant results or new studies of unclear quality.1-3 The aim of this study was to determine whether there was any change in the methodological or reporting quality of SRs in dermatology over the past decade.
This was a cross-sectional study of all SR/MAs published in 2010 and 2019 in the 10 highest-ranked dermatology journals by SCImago Journal. Methodological quality was assessed through duplicate and independent adjudication by 2 reviewers (A.M., L.K.L., J.K., A.L.) using the Risk of Bias in Systematic reviews (ROBIS) tool and, for SR/MA of interventions, the A Measurement Tool to Assess Systematic Reviews (AMSTAR) 2 tool. Reporting quality was assessed by 1 investigator (A.M.) using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and PRISMA for Abstracts (PRISMA-A) tools, and study, author, and journal characteristics were abstracted. Statistical analysis was performed with Jamovi, version 1.6.23. Methodological quality was analyzed with the χ² test for independent proportions, or the Fisher exact test if any of the expected frequencies were less than 5, and difference in proportions with 95% CIs. Reporting quality was analyzed with the independent samples t test and difference in means with 95% CIs.
In total, 21 SR/MAs from 2010 and 127 from 2019 were included There was little to no difference between 2010 and 2019 in the proportion of SR/MAs at high or unclear overall risk of bias with ROBIS or with critically low methodological quality using AMSTAR 2. The only subdomain of ROBIS with a difference in proportion of SR/MAs at high or unclear risk of bias between 2010 and 2019 was eligibility bias, with 27.3% more (95% CI, 5.42%-49.2%) in 2010 (66.7%) than 2019 (39.4%) (P = .02). There was a strong difference in proportion of PRISMA (t146 = 3.15; P = .002) and PRISMA-A (t146 = 2.46; P = .02) checklist items adequately reported between 2010 and 2019. The difference in mean proportion of PRISMA checklist items adequately reported was 3.8 items more (95% CI, 1.4-6.2 items more) in 2019 (mean [SD], 15.7 [5.1] items) than in 2010 (mean [SD], 11.9 [5.5]), and of PRISMA-A checklist items adequately reported was 0.9 items more (95% CI, 0.2-1.6 items more) in 2019 (mean [SD], 5.9 [1.5] items) than in 2010 (mean [SD], 5 [1.9] items).
While there was an improvement in the overall reporting quality of SR/MAs between 2010 and 2019, there was no improvement in the overall methodological quality as assessed by the ROBIS and AMSTAR 2 tools. This suggests a persistent potential for forming unreliable conclusions in SR/MAs through a decade of dermatological research. These results therefore highlight the urgency with which efforts to improve SR/MA methodology should be undertaken.
1. Siontis KC, Ioannidis JPA. Replication, duplication, and waste in a quarter million systematic reviews and meta-analyses. Circ Cardiovasc Qual Outcomes. 2018;11(12):e005212.
2. Katsura M, Kuriyama A, Tada M, et al. High variability in results and methodological quality among overlapping systematic reviews on the same topics in surgery: a meta-epidemiological study. Br J Surg. 2021;108(12):1521-1529.
3. Gómez-García F, Ruano J, Gay-Mimbrera J, et al. Most systematic reviews of high methodological quality on psoriasis interventions are classified as high risk of bias using ROBIS tool. J Clin Epidemiol. 2017;92:79-88.
1University of New South Wales Medicine, University of New South Wales, Sydney, New South Wales, Australia; 2National Health and Medical Research Council Clinical Trials Center, University of Sydney, Sydney, New South Wales, Australia, email@example.com
Conflict of Interest Disclosures