International Congress on
Peer Review and Scientific Publication

Haley K. Holmer,¹ Edi E. Kuhn,¹ Camber Hansen-Karr,¹ Ed Reid,¹ Mark Helfand¹

Objective

To identify methodological concerns in Agency for Healthcare Research and Quality (AHRQ) Evidence-Based Practice Center (EPC) Program reports and assess the comprehensiveness of EPC Methods Guidance using the Systematic Reviewlution (SR) Framework.¹

Design

Cross-sectional analysis of reviewer comments from 25 EPC reports submitted between June 1, 2018, and March 1, 2023. The sample included systematic reviews (n = 16), systematic review updates (n = 4), and technical briefs (n = 5). For the internal, pre–peer-reviewed phase, 1 reviewer extracted comments received from associate editors, task order officers, and AHRQ leadership. In the external peer-reviewed phase, we extracted comments from peer reviewers, key informants, technical experts, partners, and public reviewers. A senior reviewer used the SR Framework,¹ a comprehensive typology of 67 systematic review problems, to classify concerns raised in the pre–peer-reviewed and/or peer-reviewed phase and to quantify frequencies. We identified patterns in 2 areas: (1) concerns raised in the pre–peer-review phase that persisted into the peer-reviewed phase and (2) concerns missed in the pre–peer-reviewed phase that were raised in the peer-reviewed phase. Further, we determined whether problems identified in the SR Framework are addressed in EPC Methods Guidance,^2,3 and we identified methodological concerns absent from the SR Framework.

Results

Of 5024 total reviewer comments, 15% (252 of 1717) of pre–peer-reviewed comments and 10% (348 of 3307) of peer-reviewed comments addressed methodological concerns. Nearly half of the methodological concerns identified during the pre–peer-review phase pertained to grading strength of evidence (SOE) (n = 116 [46%]), followed by concerns with assessing risk of bias (n = 23 [9%]), and spin (n = 21 [8%]). The most frequent concerns raised by peer reviewers also related to spin (n = 56 [16%]), grading SOE (n = 44 [13%]), and lack of clinical expert or stakeholder perspective (n = 41 [12%]). Spin-related concerns included conclusions that went beyond the evidence, policy implications disconnected from results, failure to incorporate SOE into conclusions, and biased tone in presentation. Comments raised in the peer-reviewed phase that were not raised in pre–peer-review phase included concerns about a priori protocol decisions or inflexible methods for answering review questions. The SR Framework failed to capture several key methodologic issues identified in EPC reports, including concerns about (1) SOE ratings, (2) inadequately defined outcomes, and (3) a priori protocol registration. Over 75% (n = 51) of the 67 problems in the SR Framework were addressed in EPC methods guidance.

Conclusions

Overall, EPC reports are of high methodological quality. Notable patterns in methodological concerns emerged, with specific issues pertaining to grading SOE, spin, need for clinical expert or stakeholder perspectives, and balancing adherence to an a priori protocol against potential bias introduced by an adaptive protocol. The persistence of spin-related concerns across both review phases suggests a challenge in maintaining objectivity in evidence synthesis. Notably, the SR Framework did not capture important methodological concerns in EPC reports, suggesting potential gaps in the SR Framework for evaluating the trustworthiness of evidence synthesis products.

References

1. Uttley L, Quintana DS, Montgomery P, et al. The problems with systematic reviews: a living systematic review. J Clin Epidemiol. 2023;156:30-41. doi:10.1016/j.jclinepi.2023.01.011

2. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Page last reviewed October 2022. Effective Health Care Program, Agency for Healthcare Research and Quality. https://effectivehealthcare.ahrq.gov/products/collections/cer-methods-guide

3. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi:10.1136/bmj.n71

¹Portland VA Research Foundation, Portland, OR, haley.holmer@va.gov.

Conflict of Interest Disclosures

None of the authors have any affiliations or financial involvement that conflicts with the material presented in this abstract.

Funding/Support

This work is funded by the Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare (EHC) Program through a contract to the Scientific Resource Center (Contract No. HHSA 75Q80122C00002). The authors of this abstract are responsible for its content. Statements in the abstract do not necessarily represent the official views of or imply endorsement by the AHRQ, US Department of Health and Human Services.