Abstract
Influence of Evaluation Criteria on Overall Assessment in Peer Review of Project Grants Submitted to the Swiss National Science Foundation
Stéphanie Würth,1 Katrin Milzow,1 Matthias Egger1
Objective
The Swiss National Science Foundation (SNSF) supports basic science in all disciplines. External peer reviewers assess 3 evaluation criteria: (1) the scientific track record and expertise of applicants (track record); (2) the project’s scientific relevance, originality, and topicality (scientific relevance); and (3) the suitability of the methods and feasibility (methods and feasibility). The peer reviewers also provide an overall assessment. We aimed to identify the relative influence of the 3 evaluation criteria on the overall assessment.
Design
We analyzed reviewer ratings for 7139 project proposals submitted from 2012 to 2015. A total of 21,902 reviews were obtained from experts from more than 80 countries. The external reviewers rated the 3 evaluation criteria and the overall assessment in 6 categories: outstanding, excellent, very good, good, average, and poor. We assigned scores from 1 (poor) to 6 (outstanding) to the ratings. We performed a multivariate linear regression analysis, with the score from the overall assessment as the dependent variable and the scores from the 3 criteria as the independent variables.
Results
The mean score for the overall assessment was 4.46. Among the 3 criteria, the track record of the applicants received the highest average score (4.82), followed by scientific relevance (4.58) and the suitability of methods and feasibility (4.27). The linear multivariate regression analysis showed that the overall assessment was influenced most by the suitability of the methods and feasibility (b, 0.436; 95% CI, 0.430-0.442), followed by scientific relevance (b,0.401; 95% CI, 0.394-0.408) and the applicants’ track record (b, 0.221; 95% CI, 0.214-0.228) (Table). There were interesting differences between fields of research (humanities and social sciences, mathematics, natural and engineering sciences, and biology and medicine); the suitability of the methods and feasibility was more important in biology and medicine, whereas the track record received more weight in mathematics and natural and engineering sciences.
Conclusions
The greater weight given to the methods and feasibility and scientific relevance align with the principles of the project rather than career funding. Differences between fields of research exist and may reflect differences in research cultures.
1Swiss National Science Foundation, Bern, Switzerland, stephanie.wuerth@snf.ch
Conflict of Interest Disclosures:
Prof Egger is president of the Swiss National Science Foundation’s National Research Council.