Abstract
A Bayesian Approach to Reduce Bias in the Ranking of Peer-Reviewed Grant Proposals Submitted to the Swiss National Science Foundation
Rachel Heyard,1,2 Manuela Ott,2 Janine Bühler,2 Georgia Salanti,3 Matthias Egger2,3
Objective
Funding agencies rely on peer review to select grant proposals deserving of funding. Peer review has limitations and may be biased. The interrater reliability between reviewers and panels is low, particularly for proposals near the funding line. A method for ranking grant proposals was developed to reduce bias and accommodate the uncertainty in evaluation and funding decisions.
Design
Proposals submitted to a call were peer reviewed and later discussed in evaluation panels. After the discussion, all panel members electronically rated each proposal on a scale from 1 to 9. Panel members with conflicts of interest did not participate in the discussion or vote. Simple averages can introduce bias if panel members with distinct behaviors (eg, very critical or very generous) cannot vote. Furthermore, there may be uncertainty in ranking. A ranking approach based on a flexible bayesian hierarchical model (BHM) was developed to avoid the long and biased discussions of proposals around the funding line and was compared with standard procedure. The BHM accounted for the correlated data structure due to the same panel members voting on a set of proposals and modeled explicitly the uncertainty present at different stages of the evaluation process. As such, the BHM described the whole distribution of the rank of each proposal. The 50% credible intervals around the ranks helped assign the proposals into 3 groups: accepted, random selection, and rejected. The random selection group was composed of proposals of similar quality near the funding line. The approach was flexible and could take special cases into account. For example, if proposals were discussed in subpanels, the further level of dependency could be accounted for in the model. The convergence of the BHM was investigated using Gelman-Rubin convergence diagnostics.
Results
The approach was simulated in the Swiss National Science Foundation (SNSF) early career fellowship scheme call of February 2020. A total of 181 fellowship grants were submitted to 5 disciplinary panels—humanities (23 fellowship grants), social sciences (38), medicine (35), biology (35), and STEM (50); 79 were discussed in the panel, and 32 were funded (Figure 12). A funding line was drawn based on the available budget. A random selection group was identified for 2 panels: medicine and STEM. Applications to other funding schemes will be presented.
Conclusions
In this study, a method to address the limitations of peer review of good but not outstanding proposals was developed. The bayesian ranking approach ensured a transparent translation of votes into a ranking of the proposals. The approach was extensively discussed with stakeholders in 2021, and the Research Council of the SNSF recently decided to adopt it in most of its funding schemes later in 2022.
1Center for Reproducible Science, University of Zurich, Zurich, Switzerland, rachel.heyard@uzh.ch; 2Swiss National Science Foundation, Bern, Switzerland; 3Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
Conflict of Interest Disclosures
Matthias Egger is the president of the Research Council of the Swiss National Science Foundation. No other disclosures were reported.