Assessment of a Structured and Mentored Peer Review Curriculum on Quality of Peer Review

Ariel M. Lyons-Warren,1,2 Whitley W. Aamodt,3 Roy Strowd,4 Kathleen M. Pieper,5 José G. Merino5,6

Objective

The Resident & Fellow Section (RFS) of Neurology expanded and formalized a biannual virtual, mentored peer review training program in 2020. Similar to prior studies,¹ qualitative data demonstrated that mentored peer review improved understanding of and confidence with independent review.² The objective of the current study was to quantitatively evaluate review quality before and after a mentored peer review program for neurology residents and fellows.

Design

In this pre-post intervention study, faculty mentors chosen from a national pool of experienced reviewers with an interest in mentoring were paired with trainee peer reviewers. Mentees were selected from residents and fellows who responded to a call for program participants via American Academy of Neurology social media channels or who applied to the RFS editorial board. Mentees first completed unassisted reviews of a standardized manuscript. Participants then received structured resources on how to review a manuscript and construct written peer review. Mentor-mentee dyads completed 2 reviews over 6 months. Mentees also completed unassisted postprogram reviews of a standardized manuscript. The association between program participation and peer review quality was quantitatively assessed by 3 independent evaluators scoring the preprogram and postprogram unassisted reviews using a modified version of the Review Quality Instrument (RQI)³ comprising 4 items scored present or absent (1 or 0) and 10 items scored on a 5-point Likert scale for a maximum score of 55. Higher scores indicated better review quality. Evaluator scores were averaged for each participant, and a paired t test was used to compare preintervention and postintervention scores. P < .05 was considered significant.

Results

A total of 20 mentor-mentee pairs, including 8 incoming members of the RFS editorial board and 12 neurology trainees, were enrolled over 2 sessions in 2021. Two mentees failed to complete the postprogram unassisted review and were excluded. Total modified RQI scores ranged from 15.67 to 39.00 before the program and 19.67 to 43.67 after the program. The mean total score for all participants increased following completion of the program (preprogram, 26.38; postprogram, 31.70; P < .001). Postprogram reviews were more likely to include separate comments for editors and authors (mean preprogram score, 0.30; mean postprogram score, 0.91; P < .001), more likely to identify the strengths and weaknesses of the research methods (mean preprogram score, 2.75; mean postprogram score, 3.18; P = .02), and more likely to comment on the interpretation of results (mean preprogram score, 2.33; mean postprogram score, 2.82; P = .009). Preprogram and postprogram scores for all 14 measures of the modified RQI are shown in Table 10.

Conclusions

Peer review quality improved for residents and trainees following completion of a structured, mentored peer review curriculum. Formal mentoring to teach the proper approach to peer review is one tool to expand the bench of available quality peer reviewers.

References

1. Wong VSS, Strowd RE 3rd, Aragón-García R, et al. Mentored peer review of standardized manuscripts as a teaching tool for residents: a pilot randomized controlled multi-center study. Res Integr Peer Rev. 2017;2:6. doi:10.1186/s41073-017-0032-0 2. Lyons-Warren AM, Aamodt WW, Pieper KM, Strowd R. A sustainable, comprehensive, mentored program for teaching peer review to neurology residents. Presented at: American Academy of Neurology Annual Meeting, April 17-22, 2021. 3. van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999;52(7):625-629. doi:10.1016/s0895-4356(99)00047-5 ¹Section of Pediatric Neurology and Developmental Neuroscience, Department of Pediatrics, Baylor College of Medicine, Houston, TX, USA, lyonswar@bcm.edu; ²Jan and Dan Duncan Neurological Research Institute, Texas Children’s Hospital, Houston, TX, USA; ³Department of Neurology, University of Pennsylvania, Philadelphia, PA, USA; ⁴Wake Forest School of Medicine, Winston-Salem, NC, USA; ⁵Neurology, Pittsford, NY, USA; ⁶Department of Neurology, Georgetown University Medical Center, Washington, DC, USA

Conflict of Interest Disclosures

Ariel M. Lyons-Warren reported serving as the chair of the Resident and Fellow Section Mentored Review Program. Whitley W. Aamodt reported receiving an editorial stipend from Neurology and serving as the deputy section editor for the Resident and Fellow section of Neurology. Roy Strowd reported serving as a consultant for Monteris Medical Inc and Novocure; receiving an editorial stipend as section editor for the Resident and Fellow section of Neurology; having received stipends as an educational lecturer for Lecturio and Kaplan; and having received research and grant support from the American Academy of Neurology (AAN), the American Board of Psychiatry and Neurology, American Society for Clinical Oncology, Southeastern Brain Tumor Foundation, and Jazz Pharmaceuticals. Kathleen M. Pieper reported receiving salary support from AAN and serving as the senior managing editor of Neurology. José G. Merino reported receiving salary support from the AAN for his editorial role and serving as editor in chief of Neurology.

Acknowledgements

We thank the residents and fellows who participated in the program and completed the preprogram and postprogram assessments as well as the senior reviewers who served as mentors. In addition, we thank Seth Retzlaff in the Neurology editorial office for the handling and tracking of manuscripts. We greatly appreciate their time and commitment to improving the next generation of peer reviewers.