Abstract
Patient-Centered Outcomes Research Institute (PCORI) Methodology Standards to Improve the Design and Reporting of Research
Evan Mayo-Wilson,1 Kelly Vander Ley,2 Kay Dickersin,1 Mark Helfand2
Objective
The Patient-Centered Outcomes Research Institute (PCORI) began receiving funding applications in September 2011 and published Methodology Standards in November 2013 addressing issues related to research design and transparent reporting. PCORI requires that investigators of funded studies submit a draft final research report (DFRR) that is peer reviewed by an external team; after revision, in response to peer-review, the final report is published on the PCORI website. We sought to determine whether research described in DFRRs adheres to the PCORI standards.
Design
Research funded before the standards were published (November 2013) was eligible for the study, as were DFRRs received by February 1, 2017. Thus, the standards were not in place when eligible studies were funded but were in place when DFRRs were submitted. We excluded PCORI-funded research for “improving methods for conducting PCOR.” In 2017, we used the DFRRs to develop a 57-item checklist to assess adherence to the standards. Two authors (E.M.W. and K.V.L.) independently rated each eligible DFRR, compared disagreements, and resolved differences through discussion.
Results
Among 31 eligible DFRRs, none adhered to all standards. Nonadherence was attributed to both incomplete reporting and to nonadherence with recommendations for study design. Examples of nonadherence in 5 areas of interest include (1) formulating research questions: most reports neither included nor cited a systematic review, and most did not include or cite a study protocol; (2) patient-centeredness: all reports mentioned patient engagement but none described engaging patients in all parts of their research as specified in the standards; (3) data integrity and rigorous analyses: most reports included patient-reported outcomes but few described how patient-reported outcomes would be interpreted by patients; (4) preventing and handling missing data: many reports did not use appropriate methods for handling missing data; and (5) heterogeneity of treatment effects: most reports examined heterogeneity, commonly using subgroup analyses, but few studies were designed to conduct confirmatory tests for heterogeneity.
Conclusions
Our results identify common limitations in research funded by PCORI prior to publishing the 2013 Methodology Standards. Further studies are needed to determine whether peer review of DFRRs improves adherence to the standards in the final research reports and whether peer reviewers consider nonadherence to be related to important limitations in interpretation. Observational studies might explore whether research funded after the standards were published is better designed and better reported.
1Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA, evan.mayo-wilson@jhu.edu; 2Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, OR, USA
Conflict of Interest Disclosures:
Kay Dickersin is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. None of the authors was directly involved in any of the 31 eligible and included studies. During peer review of PCORI-funded research, the authors recused themselves from discussions about research conducted at their institutions (ie, Johns Hopkins University for Kay Dickersin and Evan Mayo-Wilson; Oregon Health & Science University for Mark Helfand and Kelly Vander Ley); these reports were included in the sample assessed by Evan Mayo-Wilson and Kelly Vander Ley for this study.
Funding/Support:
This study was conducted under a contract to provide peer review services for PCORI research reports. Mark Helfand is the principal investigator of the Peer Review of PCORI’s Funded Research, and the other authors are associate editors. All authors receive support from PCORI for reviewing research reports and for conducting research about the peer review process.
Role of the Funder/Sponsor:
PCORI was not involved in conducting this study, but the study used research reports submitted as part of the PCORI peer review process. PCORI reviewed and approved this abstract for submission.
Acknowledgments:
We thank Holly Somers and Kira Lesly, Oregon Health & Science University, for their assistance in identifying and organizing eligible research reports.