Scientific Quality in a Series of Comparative Effectiveness Research Studies

Harold Sox,1,2 Evan Mayo-Wilson,1,2 Kelly Vander Ley,1,2 Marina Broitman,1,2 David Hickam,1,2 Steven Clauser,1,2 Yen-Pin Chiang,1,2 Evelyn Whitlock1,2


Markers of high-quality comparative effectiveness research (CER) studies are largely unknown but could be valuable to funders and future applicants for CER funding. Our long-term objective is to identify variables associated with CER scientific quality and impact. The objective of this preliminary report is to describe the frequency of measures of CER study quality.


This is a case series of CER studies funded during the first funding cycle (2013) of the Patient-Centered Outcomes Research Institute (PCORI). Awardees are required to submit a final research report (FRR), which undergoes external peer review and is published on the PCORI website when the principal investigator (PI) meets revision requirements. We are using the original application to investigate study and PI-related variables potentially associated with study quality, and are assessing study quality of the peer-reviewed report using US Preventive Services Task Force (USPSTF) criteria (good, fair, poor) and adherence to PCORI methodology standards. When the case series is complete we will study associations between markers of study quality and publication outcomes (citations in published articles, systematic reviews, and practice guidelines; Altmetric scores) and between direct study quality measures and those outcomes.


Among 98 FRRs received by early June 2017, 5 have completed peer review. Candidate PI-based variables potentially associated with study quality include number of research awards from the National Institutes of Health, Agency for Healthcare Research and Quality, Centers for Disease Control and Prevention, or the Department of Veterans Affairs (mean, 1 [range, 0-5]; median, 0); number of studies in major journals as first or last author (mean, 2.8 [range, 0-8]; median, 0), the PI’s H factor (mean, 23 [range, 16-41]; median, 22), and years since he or she was granted the highest academic degree (19.2 [range, 16-21]; median, 20). All 5 studies were of “fair” quality according to USPSTF grading criteria. Each of PCORI’s 5 cross-cutting Methodology Standards (which had not been published when the studies in this report were funded) comprise several component standards (range, 7-17), and rates of meeting the standards varied from 15% (standard for managing missing data) to 34% (standard for formulating research questions). We expect to complete peer review and to report on 20 more research reports.


With short-term follow-up on a series of approximately 300 CER studies funded by PCORI through July 2019, this study may eventually provide measures of specific methodological shortcomings and variables associated with CER quality and impact.

1Patient-Centered Outcomes Research Institute, hsox@pcori.org, Johns Hopkins University, Baltimore, MD, USA; 2Oregon Health Sciences University, Portland, OR, USA

Conflict of Interest Disclosures:

None reported.