Abstract
Diversity Among Reviewers Assigned to Evaluate a Paper as a Factor in Diversifying Perspective and Improving the Peer Review Process in Computer Science
Navita Goyal,1 Ivan Stelmakh,2 Nihar B. Shah,3 Hal Daumé III1
Objective
Peer review often involves multiple reviewers to reduce bias and broaden perspectives.1,2 We examined the role of diversity among reviewers assigned to evaluate a paper as a factor in diversifying perspectives and improving the utility of the peer review process.3 We proposed 2 desiderata: reviews should cover most contents of the paper (high coverage) and reviews should add information not already present in other reviews (low redundancy). We hypothesized that reviews from diverse reviewers would exhibit higher coverage and lower redundancy.
Design
We conducted a cohort study using data from the International Conference on Machine Learning, a top-tier venue in computer science. Data were collected in February to April 2020 and analyzed January to April 2024. Reviewer diversity was defined along 5 binary dimensions: whether reviewers belonged to the same organization, same geographical region, similar seniority levels, similar research topics, or had common coauthors or papers. Organization and geographical data were obtained from reviewer profiles, seniority was based on h-index, coauthorship was determined from Google Scholar publications records, and topics were inferred using a topic model applied to the abstracts of reviewers’ prior publications. Seniority and topical diversity were binarized using median h-index and topical similarity scores, respectively. Coverage was measured by the breadth of review criteria addressed—specifically, the aspects of the paper discussed (eg, summary, motivation, originality, soundness) and the types of arguments used (eg, fact, request, reference, quote)—and lexical and semantic overlap between the reviews and the paper abstract. Redundancy was measured as the lexical and semantic overlap among different reviews. Both measures were computed using natural language processing tools. To isolate the relationship between diversity and review coverage and redundancy, we controlled for potential confounders, including reviewers’ expertise (measured as similarity between reviewer’s published papers and the submitted paper), reviewer characteristics (organization, location, topic, coauthors, and seniority), and paper content (by comparing diverse and nondiverse slates of reviewers for the same paper). We used a weighted linear regression model with review coverage or redundancy as the dependent variable and diversity and confounding factors as independent variables. Statistical significance was assessed using t tests.
Results
This study included 4991 submitted papers and 3637 reviewers. Diversity in publication networks and seniority were associated with broader coverage of review criteria (Table 25-1066). Topical diversity was associated with a broader coverage of the paper. No significant association was found between coverage and organizational and geographical diversity. Except for geographical diversity, diversity in organizations, seniority, topics, or publications networks were associated with lower redundancy among reviews. Furthermore, publication network-based diversity alone was associated with varying perspectives (that is, low redundancy) within specific review criteria.
Conclusions
Our study revealed that, in addition to optimizing individual reviewer expertise, choosing diverse slates of reviewers can lead to better reviews with higher coverage and lower redundancy. The findings are limited to certain aspects of review utility, namely redundancy and coverage, operationalized through specific measures. We recognize that there may be other factors that are crucial, either generally or specific to a particular conference or journal, which we have not addressed.
References
1. Shah NB. Challenges, experiments, and computational solutions in peer review. Comm ACM. 2022;65(6):76-87. doi:10.1145/3528086
2. Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer review. J Am Soc Inf Sci Technol. 2013;64(1):2-17. doi: 10.1002/asi.22784
3. Jackson SE, May KE, Whitney K, Guzzo RA, Salas E. Understanding the dynamics of diversity in decision making teams. Team Effectiveness Decision Making Organ. 1995;204:261.
1University of Maryland, College Park, MD, US, navita@umd.edu; 2New Economic School, Moscow, Russia; 3Carnegie Mellon University, Pittsburgh, PA, US.
Conflict of Interest Disclosures
Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.
Funding/Support
This research was supported by National Science Foundation award 1942124 and Office of Naval Research award N000142212181.
Role of the Funder/Sponsor
The funders played no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.
Additional Information
We thank the current and former members of the University of Maryland Computational Linguistics and Information Processing laboratory, especially Alexander Hoyle and Connor Baumler, for their useful suggestions and feedback on this work. This study was approved by the University of Maryland Institutional Review Board.