Abstract
Construction and Validation of Instruments for the Peer Review of Grant Proposals in Peru
Max Carlos Ramírez-Soto,1 Laura Alvarado-Barbarán,1 Dianeth Rojas-Naccha,1 Ayda Luna-Mercado,1 Arlet Arce-Zavala1
Objective
Peer review plays a central role in selecting research proposals to obtain grants from science, technology, and innovation (STI) agencies.1 Different criteria and questions are used during this process.2 Given the time and resources involved, there is no available information on the reliability of the tools and review criteria used in the peer review of grant proposals. Peru’s National Program for Scientific Research and Advanced Studies (PROCIENCIA) uses a 2-tiered review process: an initial peer review to assess scientific merit, followed by a panel review to evaluate proposal relevance in line with funding priorities. Expert scientists are involved in both stages of the review process. Proposals may be improved prior to the final allocation of funding. In that context, we assess the reliability of the instruments used for the peer review of grant proposals at PROCIENCIA.
Design
This cross-sectional study assessed the construction and validation of 7 instruments used by PROCIENCIA for the peer review of grant applications between January 2022 and December 2024. Peer review instruments were applied in 7 funding calls (Figure 25-1021). Each financial instrument has distinct requirements and aims. The process of constructing and validating the instruments was carried out in the following stages: (1) literature review, (2) instrument construction, and (3) content validation. The literature search included studies on the criteria for reviewing STI grant applications, including the review criteria used by STI agencies. The findings from the literature review were used as evidence for the development of the evaluation criteria and questions included in the peer review instruments. We used a modified Delphi process to validate the content,3 which included a review round by 8 to 15 expert panelists for each instrument. The experts were senior researchers with extensive experience in reviewing grant applications and scientific articles in various fields of scientific and technological knowledge. Each instrument contained 4 to 7 criteria (eg, knowledge of the researchers, relevance and coherence of the proposal, the scientific-technological aspect of the proposal, feasibility, results, and budget) as well as 3 to 8 questions. Content validation was performed at the overall instrument level (overall score) and for each criterion (individual score). Cronbach α was used to assess internal consistency; a value greater than .75 was considered acceptable.
Results
The 7 instruments used for the peer review of grant applications had an overall reliability index between .86 and .98 (Figure 25-1021). All instruments had acceptable reliability indices regarding assessment criteria (>.75). The assessment criteria of the doctoral program, technological development, and improvement of laboratory equipment instruments had a higher reliability index (range, .89-1.00 and .92-1.00, respectively).
Conclusions
Our results show that PROCIENCIA uses validated and reliable instruments for the peer review of grant proposals. Validation and reliability of peer review instruments for STI funding is helpful in ensuring the integrity of the review process and in selecting research proposals for funding. Therefore, funding agencies should consider validating peer review instruments.
References
1. Hug SE, Aeschbach M. Criteria for assessing grant applications: a systematic review. Palgrave Commun. 2020;6(1):1-15. doi:10.1057/s41599-020-0412-9
2. Abdoul H, Perrey C, Amiel P, et al. Peer review of grant applications: criteria used and qualitative study of reviewer practices. PLoS One. 2012;7(9):e46054. doi:10.1371/journal.pone.0046054
3. Hsu C, Sandford BA. The Delphi technique: making sense of consensus. Practl Assess Res Eval. 2007;12(1):10. doi:10.7275/pdz9-th90
1Beneficiary Selection Sub-Unit, National Program of Scientific Research and Advanced Studies (PROCIENCIA), National Council for Science, Technology and Innovation (CONCYTEC), Lima, Peru, mramirez@prociencia.gob.pe.
Conflict of Interest Disclosures
None reported.
Acknowledgment
The authors would like to thank Sixto Sánchez, president of CONCYTEC, and Dora Blitchtein Winicki De Levy, executive director of PROCIENCIA, for the technical support for the study.