Prevalence of High or Unclear Risk of Bias Assessments in Diagnostic Accuracy Studies Included in Cochrane Reviews

Nicola Di Girolamo,1 Reint Meursinge Reynders,2 Alexandra Winter3,4


The validity of systematic reviews of diagnostic accuracy is dependent on the extent of bias in included primary studies. The objective of this study was to assess the risk of bias in primary studies of diagnostic accuracy included in Cochrane reviews according to the Quality Assessment for Studies of Diagnostic Accuracy 2 (QUADAS-2) tool.


All systematic reviews of diagnostic accuracy published in the Cochrane database in 2016 that used the QUADAS-2 quality assessment tool and that reported results with the risk of bias figure were eligible. The primary outcome was the prevalence of high or unclear risk-of-bias scores for the 4 QUADAS-2 domains—patient selection, index test, reference standard, and flow and timing—among the primary studies in these diagnostic accuracy reviews. Two investigators selected the eligible reviews and assessed the risk of bias scores for primary studies. Disagreements were resolved by consensus.


Of 46 eligible systematic reviews, 35 were included, 6 were excluded because they did not use the QUADAS-2 tool, 4 were excluded because they lacked the risk-of-bias figure, and 1 review was a duplicate. A total of 1045 primary studies with 4133 bias assessments were identified; 1044 reported risk of bias for patient selection, 1002 for index test, 1044 for reference standard, and 1043 for flow and timing. A total of 2319 of 4133 domains (56%) were assessed as having high or unclear risk of bias, with 1814 (44%) having low risk of bias. For all domains except flow and timing, the majority of outcomes were scored as having high or unclear risk of bias.


Primary studies in systematic reviews of diagnostic accuracy are often rated as having high or unclear risk of bias by QUADAS-2 criteria. Inclusion of such studies in systematic reviews and meta-analyses may jeopardize final results and interpretation. This study is limited by the lack of accounting for clustering of risk-of-bias assessments within primary studies evaluated in systematic reviews and for the exclusion of non-Cochrane reviews, which may limit the generalizability of the findings. Although further investigation is indicated to evaluate whether additional training of systematic reviewers would decrease the prevalence of high and unclear risk of bias, the findings point to a need to improve the conduct and reporting of diagnostic accuracy studies.

1EBMVet, Cremona, Italy, nicoladiggi@gmail.com; 2Department of Oral and Maxillofacial Surgery, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands; 3Journal of the American Veterinary Medical Association, Schaumburg, IL, USA; 4American Veterinary Medical Association, Schaumburg, IL, USA

Conflict of Interest Disclosures:

None reported.


This study was self-funded.