Frequency of Citation of Clinical Trials With a High Risk of Bias

Priyanka Desai,1 Mary Butler,1 Robert L. Kane1


To compare citation rates of high, medium, or low risk of bias (ROB) clinical trials included in systematic reviews.


We used a convenience sample of systematic reviews published between 2011 and 2014 by 4 journals, the Cochrane Collaboration, and the Agency for Healthcare Research and Quality Evidence-based Practice Centers to identify a sample of clinical trials. We abstracted the trial ROB assigned by review authors, or, for trials rated as having an unclear ROB, we assigned ROB ratings using the Cochrane ROB tool. We converted quality and numeric ratings to ROB ratings and assigned overall ratings to studies that were assessed with only domains of the Cochrane Tool. We used Google Scholar as a source for citation rates and Journal Citation Reports as a source for citing the Impact Factors of journals. We reported mean trial citation rates by low, medium, and high ROB and calculated Spearman correlation coefficients to assess the association between ROB and citations and citations and impact factors. We reported findings for cited trials overall and stratified by the type of intervention .


Of 76 systematic reviews, 55 reported sufficient ROB information. Of 1456 trials with ROB ratings, 34.4% (500) were rated with low, 43.1% (628) with medium, and 22.5% (328) with high ROB. Across all intervention categories, low ROB studies were more frequently cited than high ROB studies (Table). The correlation between ROB and citation rates was generally weak but statistically significant for trials of device, lifestyle, and pharmaceutical interventions. Across all intervention categories, there was a moderate correlation between citation rates and Impact Factors (correlation coefficients from 0.46 to 0.62, P < .01)


Weak correlations between clinical trial ROB and citation rates suggest that ROB is not a meaningful factor in the decision to cite a publication. We did not examine the context in which high ROB trials were cited; however, their inclusion in systematic reviews may pose a risk to evidence-based practice when ROB is not fully considered. High ROB ratings are difficult to interpret, and a variety of factors may contribute to the rating. The field should consider how to adequately inform readers and end users about the potential biases of published clinical trials.

1University of Minnesota School of Public Health, Minneapolis, MN, USA, butl0092@umn.edu

Conflict of Interest Disclosures:

None reported.


None reported.


Dr Kane passed away on March 6, 2017.