Abstract
Methodological and Reporting Quality of Systematic Reviews Underpinning Clinical Practice Guidelines
Cole Wayant,1 Matt Vassar1
Objective
This study summarizes the findings of 3 separate studies conducted simultaneously to determine the methodological and reporting quality of systematic reviews (SRs) underpinning clinical practice guidelines (CPGs) in pediatric obesity, opioid use disorder, and ST-elevated myocardial infarction.
Design
A search of guideline clearinghouse and professional organization websites was conducted for guidelines published by national or professional organizations. We included all reviews cited by authors of CPG, including Cochrane reviews, and removed duplicates prior to data extraction. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) and AMSTAR (A Measurement Tool to Assess Systematic Reviews) instruments were used to score SRs and meta-analyses cited in CPGs. PRISMA and AMSTAR are validated tools for measuring reporting quality and methodological quality, respectively.
Results
The mean PRISMA total scores for the pediatric obesity, opioid use disorder, and ST-elevated myocardial infarction SRs across all CPGs were 16.9, 20.8, and 20.8, respectively. The mean AMSTAR total scores were 4.4, 8.8, and 6.1, respectively. Consistently underreported items on the PRISMA checklist were items 5 (protocol registration), 8 (search strategy), 15 (risk of bias for cumulative evidence), and 22 (risk of bias across studies). Consistently underreported items on the AMSTAR checklist were items 4 (duplicate extraction/validation), 5 (list of included/excluded studies), 8 (quality of evidence assessments), 10 (publication bias assessments), and 11 (conflict of interest disclosure). Altogether, our study included 150 SRs and 29 CPGs, with only 9 CPGs assigning grades to their recommendations. The 150 SRs were cited a total of 308 times: 95 times as a direct evidence for graded recommendations, 21 times as direct evidence for nongraded recommendations, 189 times as supporting evidence, and 3 times for unclear reasons.
Conclusions
These investigations into CPGs in pediatric obesity, opioid use disorder, and ST-elevated myocardial infarction revealed a consistent lack of overall methodological and reporting quality in the included SRs as well as heterogeneity in the use of grading scales, or lack thereof. Because SRs are considered by most to be level 1A evidence, an apparent lack of quality may impair clinical decision making and hinder the practice of evidence-based medicine. Items such as PRISMA items 15 and 22 and AMSTAR items 10 and 11 are of particular concern because these items ensure that bias assessments are performed and conflicts of interest are disclosed.
1Oklahoma State University Center for Health Sciences, Department of Analytical and Institutional Research, Tulsa, OK, USA, cole.wayant@okstate.edu
Conflict of Interest Disclosures:
None reported.
Additional Contributors:
We acknowledge the work done by the members of the team that conducted the investigations into opioid use disorder (Andrew Ross and Justin Rankin) and ST-elevated myocardial infarction (Jared Scott and Ben Howard) as well as all of the faculty and staff at the Oklahoma State University Center for Health Sciences and Oklahoma State University Medical Center that assisted in the process of completing the investigations.