Laura P. Forsythe,1 Lori B. Frank,1 A. Tsahai Tafari,1 Sarah S. Cohen,2 Michael Lauer,1,3 Steve Clauser,1 Christine Goertz,1,4 Suzanne Schrandt1
Scientific review of funding applications was established to fund rigorous, high-impact research. The Patient-Centered Outcomes Research Institute (PCORI) uses unique review criteria and includes patients and other healthcare stakeholders as reviewers. This study assesses the relative importance of each criterion and the associations of different reviewer types’ ratings with final scores and funding outcomes.
This study is a retrospective, cross-sectional analysis of PCORI Merit Review administrative data for 5 funding cycles from 2013 through 2015.Before a panel discussion, patients and other stakeholders were required to score each application overall and on 3 criteria: potential to improve care and outcomes, patient-centeredness, and engagement. Scientist reviewers also scored impact of condition and technical merit. Scores ranged from 1 (exceptional) to 9 (poor). All reviewers provided postdiscussion overall scores. Funding decisions were made by the PCORI Board of Governors based on Merit Review, portfolio balance, and programmatic fit. Linear regression models stratified by reviewer type (ie, scientist, patient, or other stakeholder) tested associations of postdiscussion overall scores with prediscussion criteria scores. Associations between funding decisions and prediscussion criteria scores were tested using logistic regression. All models adjusted for funding program, review cycle, and principal investigator characteristics (ie, National Institutes of Health funding, clinical degree[s] of applicants, and years of experience of applicants).
A total of 535 reviewers (254 scientists, 139 patients, and 142 stakeholders) reviewed 1312 applications; 663 (50.5%) were discussed and 121 (9.2%) were funded. Prediscussion mean (SD) overall scores were higher (ie, worse) for scientist reviewers (4.9 [2.1]) than patient reviewers (4.2 [2.2]) and stakeholder reviewers (4.2 [2.1]) (P < .001). The mean overall score postdiscussion was 28.0 for funded applications and 50.1 for unfunded applications. All reviewer types changed their overall score through panel discussion for more than half of the applications. Score agreement across reviewer types was greater postdiscussion. For all reviewer types, postdiscussion review scores were positively associated with at least 1 prediscussion criterion score from each of the 3 reviewer types (Table). The strongest association with postdiscussion overall scores for all reviewer types was scientists’ ratings of technical merit. More favorable prediscussion ratings by each reviewer type for the potential to improve care and outcomes and scientist reviewers’ ratings of technical merit and patient-centeredness were associated with greater likelihood of funding.
Scientist, patient, and stakeholder views of applications converged following discussion. Technical merit is critical to funding success, but patient and stakeholder ratings of other criteria also relate to application disposition. Results suggest that research application review can incorporate nonscientist perspectives in scoring and funding outcomes.
1Patient-Centered Outcomes Research Institute, Washington, DC, USA, firstname.lastname@example.org; 2EpidStat Institute, Ann Arbor, MI, USA; 3National Institutes of Health, Bethesda, MD, USA; 4Palmer Center for Chiropractic Research, Davenport, IA, USA
Conflict of Interest Disclosures:
This work was funded by the Patient-Centered Outcomes Research Institute (PCORI).
Role of the Funder/Sponsor:
Members of the PCORI staff, Board of Governors, and Methodology Committee designed and conducted the study and reported the results.