Abstract
Assessment of Grant Peer Reviewers’ Tolerance for Risk in Research Proposals
Stephen A. Gallo,1 Karen B. Schmaling2
Objective
Grant peer review relies on scientists’ ability to evaluate research proposals’ quality. Such judgments are sometimes beyond reviewers’ discriminatory power, leading to reliance on subjective biases, including preferences for lower-risk, incremental projects.1,2 However, reviewers’ research risk tolerance during grant peer review has not been well studied.
Design
In late 2020 to early 2021, participants were recruited through email invitation using the American Institute of Biological Sciences’ reviewer databases and the National Institutes of Health’s panel rosters. A cross-sectional, prospective study was conducted of peer reviewers’ numeric evaluations of mock primary reviewers’ comments of a theoretical research proposal in which the level and sources of risks and weaknesses were manipulated. Specifically, there were 4 scenarios, with risks associated with either the investigator (limited experience in leading independent research; n = 199), the approach (lacking pilot studies; n = 205), both investigator and approach (n = 201), or neither (control; n = 605). Each participant evaluated the control and 1 randomly chosen manipulated risk scenario (the order of presentation was randomized). Risk tolerance was measured by the 13-item Openness to Experience scale from the NEO Five-Factor Inventory 3.3
Results
Violin plots of overall and criteria scoring data from 605 participants (lower scores are more positive) showed that the experimental manipulation was effective: Scenarios with manipulated risk received more mediocre overall scores and more dispersed scoring distributions than the control scenario (Figure 32). Ordinal logistic regression models (Nagelkerke R2 = 0.78) suggest that the manipulated risks in these scenarios (compared with control) were strongly associated with participants’ overall scores, with odds ratios of 15.0 (95% CI, 3.4-66.0) for the investigator risk scenario, 21.7 (95% CI, 5.5-85.4) for the approach risk scenario, and 36.2 (95% CI, 8.5-154.4) for the scenario with both risks. The approach criterion score was associated with the overall score for all scenarios; significance and innovation criteria scores had stronger associations with overall scores in the control scenario. Openness to experience was not associated with scores, but differing levels of reviewer-reported scoring leniency were observed.
Conclusions
These data suggest that the evaluation of risks dominates reviewers’ evaluation of research proposals and is an important source of interreviewer variability. A weakness is that this study did not involve actual grant proposals. Training peer reviewers to consider risks as potential assets (eg, innovation) may represent an area of useful intervention for science advancement.
References
1. Lamont M. How Professors Think: Inside the Curious World of Academic Judgment. Harvard University Press; 2009.
2. Luukkonen T. Conservatism and risk-taking in peer review: emerging ERC practices. Res Eval. 2012;21(1):48-60. doi:10.1093/reseval/rvs001
3. McCrae RR, Costa PT. NEO Inventories: Professional Manual. PAR, Inc; 2010.
1Scientific Peer Advisory and Review Services Division, American Institute of Biological Sciences, Herndon, VA, USA, sgallo@aibs.org; 2Department of Psychology, Washington State University, Vancouver, WA, USA
Conflict of Interest Disclosures
None reported.
Funding/Support
This work was supported by the Social, Behavioral, and Economic Sciences Office of Multidisciplinary Activities at the National Science Foundation under the standard grant award numbers 1951132 and 1951251.