Abstract
Development of the Quality Assessment of Prognostic Accuracy Studies (QUAPAS) Tool for Assessing Risk of Bias in Prognostic Accuracy Studies
Jenny Lee,1 Frits Mulder,2 Mariska Leeflang,1 Robert Wolff,3 Penny Whiting,4 Patrick M. Bossuyt1
Objective
Systematic reviews of prognostic accuracy studies often use the QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies) tool1 to assess risk of bias and applicability of included studies because no comparable instrument exists for prognostic accuracy studies. The QUAPAS (Quality Assessment of Prognostic Accuracy Studies) tool is an adaptation of QUADAS-2 for prognostic accuracy studies.
Design
Six experienced reviewers and methodologists in the area of test evaluation and/or development of risk of bias tools participated in the development of QUAPAS. They used risk of bias tools suggested for systematic reviews by the Cochrane Prognosis Methods Group and Diagnostic Test Accuracy Working Group. Using QUADAS-2 as a starting point, domains and signaling questions from QUIPS (Quality in Prognosis Studies)2 and PROBAST (Prediction Model Risk of Bias Assessment Tool)3 were evaluated in parallel to collate a unique list of signaling questions for each domain. Additional signaling questions based on relevant sources of bias were included. The steering group of 6 experts conducted and reviewed 3 rounds of modifications before arriving at the final set of domains and signaling questions. The authors further shared QUAPAS with and invited written feedback from 10 researchers and potential end users.
Results
QUAPAS was developed to be used as QUADAS-2: specify the review question, tailor the tool, draw a flow diagram, judge risk of bias, and identify applicability concerns. Risk of bias was judged across 5 domains: participants, index test, outcome, flow and timing, and analysis (Table 54). Signaling questions assisted the final judgment for each domain. Applicability concerns were assessed for the first 4 domains. QUAPAS was used in parallel with QUADAS-2 and QUIPS in a systematic review evaluating the accuracy of a noninvasive liver test in prognosis of fibrosis and mortality. Use of QUAPAS improved the risk assessment of the flow and timing domain by the addition of a new signaling question and flagged studies at risk of bias in the new analysis domain. Judgment of risk of bias in the analysis domain was found challenging owing to sparse reporting of statistical methods.
Conclusions
QUAPAS may provide future systematic reviewers and readers with a reliable tool that can assess bias and applicability concerns in prognostic accuracy studies. Use of a systematically tailored tool will hopefully improve the quality assessment process and help produce more robust evidence base for prognostic tests.
References
1. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529-536. doi:10.7326/0003-4819-155-8-201110180-00009
2. Hayden JA, van der Windt DA, Cartwright JL, Côté P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013;158(4):280-286. doi:10.7326/0003-4819-158-4-201302190-00009
3. Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170(1):51-58. doi:10.7326/M18-1376
1Department of Epidemiology and Data Science, Amsterdam UMC, Amsterdam, the Netherlands, j.a.lee@amsterdamumc.nl; 2Department of Vascular Medicine, Cardiovascular Sciences, Amsterdam, the Netherlands; 3Kleijnen Systematic Reviews Ltd, Escrick, UK; 4Bristol Medical School, University of Bristol, Bristol, UK
Conflict of Interest Disclosures
Patrick M. Bossuyt is an advisory board member of the Peer Review Congress but was not involved in the review or decision of this abstract. No other disclosures were reported.
Funding/Support
The LITMUS project received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement 777377. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation program and EFPIA.
Role of the Funder/Sponsor
The funders had no role in the study.
Acknowledgment
The authors thank Jill Hayden, PhD (Dalhousie Medical School), for her valuable contributions to the earlier stages of developing QUAPAS. They also thank the volunteers for reviewing the tool.