Peer Review Exercises to Enhance Trainees’ Readiness to Confront Unfair or Biased Reviews
Abstract
Franki Y. H. Kung,1 Mariam Aly,2 Shahana Ansari,3 Eliana Colunga,4 M. J. Crockett,5,6 Amanda B. Diekman,3 Pablo Gomez,7 Paul C. McKee,8 Miriam Pérez,9 Sarah M. Stilwell,10 Matthew Goldrick11,12
Objective
Accumulating data suggest peer reviews can exhibit bias (ie, systematically devaluing certain topics, methods, or the study of nondominant populations1) and can be unfair with respect to process (eg, decisions are made inconsistently) or people (eg, authors are disrespected).2 Recent studies have shown that biases and unfair behaviors in peer review tend to undermine the self-reported experiences and career progression of researchers, including those from underserved populations, such as women, nonbinary individuals, and people from racial and ethnic groups other than White.1-3 This study examined whether peer review exercises increased psychology and neuroscience trainees’ perceived readiness to respond to biased and unfair reviewers.
Design
Single-session synchronous online trainings (conducted during 2023-2024) allowed for trainer-trainee and trainee-trainee interactions. Trainees were recruited for this study through social media, targeted academic organizations, and graduate programs. Trainers (professors and a postdoctoral fellow with multiple years of peer review experience) reviewed processes occurring between journal submission and receipt of a decision letter. Trainees were then given a simplified, anonymized decision letter and reviews. Exercises and discussions illustrated how trainees could plan manuscript revisions and compose responses to critiques. Finally, trainees were given examples of biased and unfair reviews, along with strategies for responding to them while acknowledging the power dynamics inherent to review. Discussion included how to maintain motivation when encountering such reviews. Participants were asked to complete surveys before and after online training in which they rated their ability to perform the following targeted skills on a scale from 1 (very little skill) to 5 (great deal of skill): define and identify bias in peer reviews; define and identify unfair language in peer reviews; respond to reviews using fair language; respond to biased or unfair reviews; and promote an inclusive and equitable peer-review culture.
Results
A total of 64 participants (42 PhD students [66%], 10 predoctoral students [16%], 8 postdoctoral trainees [13%], and 4 other trainees [5%]) rated their level of skill for 5 targeted skills. Ordinal regressions showed participants had significantly higher ratings for all skills at posttest. For example, the mean (SD) self-ratings for the skill to respond to biased on unfair reviews increased from 2.0 (1.01) on the pretest to 4.09 (0.89) on the posttest (Table 25-0893).
Conclusions
After these peer review exercises, psychology and neuroscience trainees had an increase in their perceived readiness to confront biased and unfair reviews. Work is ongoing to scale up this training using on-demand materials and further assess training impact, including measures beyond trainee self-reports.
References
1. Rogers LO, Moffitt U, McLean KC, Syed M. Research as resistance: naming and dismantling the master narrative of “good” science. Am Psychol. 2024;79:484-496. doi:10.1037/amp0001246
2. Silbiger NJ, Stubler AD. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. PeerJ. 2019;7:e8247. doi:10.7717/peerj.8247
3. Aly M, Colunga E, Crockett MJ, et al. Changing the culture of peer review for a more inclusive and equitable psychological science. J Exp Psychol Gen. 2023;152:3546-3565. doi:10.1037/xge0001461
1Department of Psychological Sciences, Purdue University, West Lafayette, IN, US; 2Department of Psychology, University of California Berkeley, Berkeley, CA, US; 3Department of Psychological and Brain Sciences, Indiana University Bloomington, Bloomington, IN, US; 4Department of Psychology and Neuroscience, University of Colorado Boulder, City, CO, US; 5Department of Psychology, Princeton University, Princeton, NJ, US; 6Department of Psychology, University Center for Human Values, Princeton University, Princeton, NJ, US; 7Department of Psychology, Skidmore College, Saratoga Springs, NY, US; 8Department of Psychology and Neuroscience, Duke University, Durham, NC, US; 9Department of Psychology, North Park University, Chicago, IL, US; 10Department of Health Behavior and Health Equity, University of Michigan, Ann Arbor, MI, US; 11Department of Linguistics, Northwestern University, Evanston, IL, US, matt-goldrick@northwestern.edu; 12Cognitive Science Program, Northwestern University, Evanston, IL, US.
Conflict of Interest Disclosures
None reported.
Funding/Support
This work was supported by grants from the National Science Foundation (DGE-2224777, DGE-2224779, DGE-2436430).
Role of the Funder/Sponsor
The funder of the study had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.
Additional Information
Training materials are freely available (https://osf.io/658ev).