Abstract
Assessment of Performance of Grant Peer Reviewers in the Canadian Health Research Funding System, 2019-2021
Clare L. Ardern,1,2 Nadia Martino,3 Sammy Nag,3 Adrian Mota,4 Karim M. Khan1,5,6
Objective
Funders must make evidence-informed decisions about how best to continue to deliver fair, equitable, and inclusive (ie, high-quality) grant peer review. The Canadian Institutes of Health Research (CIHR) commenced a quality assurance program in fall 2019 to routinely monitor the quality of peer reviewer participation in and contributions to its Project Grant Competition Peer Review Panels. The aim was to describe the performance of grant peer reviewers in the Canadian health research funding system based on assessments made by Peer Review Panel leaders.
Design
All Peer Review Panel chairs and scientific officers who led peer review for the CIHR Project Grant Competition rounds in 2019 (fall), 2020 (fall), and 2021 (spring and fall) completed CIHR’s reviewer quality feedback form immediately after the Peer Review Panel meeting and returned the form to CIHR College of Reviewers staff. The form addressed 15 elements to characterize quality peer review across 5 domains (Table 65). All Peer Review Panel members consented to anonymous assessment of their performance by the chair or scientific officer of the panel. Chairs and scientific officers, supported by CIHR staff, used the feedback form1 (5-7 minutes to complete per panel) to (1) assess reviews for appropriateness, robustness, and utility; (2) judge whether reviewer participation was professional, responsive, and engaged; and (3) identify peer reviewers with potential to serve as future chairs, scientific officers, or peer reviewer mentors.
Results
The performance of 4438 peer reviewers (1828 female reviewers [41%], 2601 male reviewers [59%], and 9 reviewers [0.2%] who did not declare sex; 2459 unique peer reviewers participating across 4 competition rounds) was evaluated by 478 chairs and scientific officers. There were between 57 and 61 panels and approximately 1000 peer reviewers in each competition round. There were 1190 reviewers participating in 1 Project Grant Competition round, 698 reviewers participating in 2 rounds, 434 reviewers participating in 3 rounds, and 139 reviewers participating in all 4 rounds. Peer review for Project Grant Competitions in 2020 and 2021 was delivered online via Microsoft Teams. Approximately 1 in 3 peer reviewers was considered to have submitted outstanding reviews or participated constructively in discussions of additional applications not assigned to that reviewer (Table 65). At least 1 in 10 peer reviewers demonstrated potential as a future chair, scientific officer, or peer reviewer mentor (Table 65). At most, 1 in 20 peer reviewers was considered to have not performed adequately with respect to review quality, participation, or responsiveness.
Conclusions
The quality of peer review for Project Grant Competitions at Canada’s health research funding agency (as assessed with CIHR’s Reviewer Quality Feedback Form) was consistent during the study period. Among peer reviewers, approximately 95% met the standard expected by chairs and scientific officers for review quality, participation, and responsiveness.
Reference
1. Canadian Institutes of Health Research. Reviewer quality feedback form. Accessed June 24, 2022. https://cihr-irsc.gc.ca/e/51654.html
1Department of Family Practice, University of British Columbia, Vancouver, BC, Canada, clare.ardern@ubc.ca; 2Sport and Exercise Medicine Research Centre, La Trobe University, Melbourne, Australia; 3Canadian Institutes of Health Research College of Reviewers, Ottawa, ON, Canada; 4Canadian Institutes of Health Research, Research Programs-Operations, Ottawa, ON, Canada; 5School of Kinesiology, University of British Columbia, Vancouver, BC, Canada; 6Canadian Institutes of Health Research–Institute of Musculoskeletal Health and Arthritis, Vancouver, BC, Canada
Funding/Support
This work was supported by a Canadian Institutes of Health Research (CIHR) Research Operating Grant (Scientific Directors) held by Karim M. Khan.
Role of the Funder/Sponsor
The funder (CIHR) had no role in the design and conduct of the study. CIHR’s College of Reviewers coordinated data collection, management, and analysis as part of its mandate to foster and deliver high-quality peer review for health research in Canada. CIHR did not participate in interpretation of the data or the preparation of the abstract; CIHR reviewed and approved the abstract prior to submission. CIHR did not participate in the decision to submit the abstract for presentation.
Conflict of Interest Disclosures
Nadia Martino (executive director, CIHR College of Reviewers), Sammy Nag (senior advisor, CIHR College of Reviewers), and Adrian Mota (associate vice president, CIHR Research Programs–Operations) are employees of CIHR. Nadia Martino and Sammy Nag are responsible for coordinating the Research Quality Assurance program for CIHR. Karim M. Khan is the CIHR–Institute of Musculoskeletal Health and Arthritis scientific director.
Additional Contributions
Zakirullah Sanaye (advisor, CIHR College of Reviewers), Nicole Haddadian (lead, CIHR College of Reviewers), and Simon Partridge (senior advisor, CIHR College of Reviewers) constructed the project database and helped produce the summary data.
Additional Information
Karim M. Khan is a co–corresponding author.