Abstract
A Computational Method to Address Strategic Behavior in Peer Review
Komal Dhull,1 Steven Jecmen,1 Pravesh Kothari,1 Nihar B. Shah1
Objective
In many peer review processes, such as grant proposal review and conference paper review in computer science, only a fixed number of submissions are accepted. Moreover, many authors are also tasked with reviewing other submissions. This is known to lead to strategic behavior, whereby reviewers manipulate the reviews they provide to increase the chances of their own submissions getting accepted.1,2 The objective of this work was to prevent such unethical behavior in peer review.
Design
New computational methods to address this problem were developed. The methods build on a prior method for strategy-proof reviewer assignment by Alon et al.3 Their method randomly partitions reviewers into 2 groups and assigns reviewers to review papers authored by reviewers in the other group. Their method then accepts for publication an equal number of papers from each group, thus guaranteeing that no reviewer can influence the outcome of their own papers by manipulating the reviews they provide (ie, “strategyproofness”). The methods proposed in the present work more carefully choose the partition of reviewers to maximize an assignment quality objective, while still satisfying strategyproofness. Large venues frequently consider such an assignment quality objective when using artificial intelligence to assign reviewers.2 The assignment procedure first computes a similarity score between every reviewer-paper pair as a proxy for assignment quality and then assigns reviewers to papers in a manner that maximizes the cumulative similarity score of the assigned reviewer-paper pairs subject to load constraints. The proposed methods aim to choose the highest-similarity assignment subject to the strategyproofness guarantee. The strategyproofness constraint could reduce the cumulative similarity score of the assignment, and this metric was empirically evaluated. This evaluation was performed on data from the International Conference on Representation Learning 2018, a top conference in artificial intelligence that reviewed 911 full papers and was a terminal venue of publication. The optimal cumulative similarity score of the assignment in the absence of strategyproofness was computed and compared with that obtained under the proposed algorithms, as well as the aforementioned baseline algorithm.3
Results
Figure 25 displays the reduction in the cumulative similarity of the assignment produced by the proposed algorithms. For reviewer and paper loads of 1, 2, and 3, respectively, the cycle-breaking algorithm lost only 3.27%, 4.73%, and 5.80% of optimal similarity; the coloring algorithm lost 11.1%, 11.9%, and 11.7% of optimal similarity; and the baseline random algorithm lost 19.0%, 17.1%, and 15.9% of optimal similarity. The similarity loss of the cycle-breaking algorithm was at least 2.5 times less than that of the random algorithm.
Conclusions
The proposed methods realized strategyproofness without a large reduction in cumulative similarity, indicating that assignment quality remained high.
References
1. Balietti S, Robert G, Dirk H. Peer review and competition in the Art Exhibition Game. Proc Nat Acad Sci. 2016;113(30):8414-8419. doi:10.1073/pnas.1603723113
2. Shah N. An overview of challenges, experiments, and computational solutions in peer review. Communications of the Association for Computing Machinery. Posted online July 7, 2022. https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf
3. Alon N, Fischer F, Procaccia A, Tennenholtz M. Sum of us: strategyproof selection from the selectors. In: Proceedings of the 13th Conference on Theoretical Aspects of Rationality and Knowledge, 2011. doi.org/10.1145/2000378.2000390
1Carnegie Mellon University, Pittsburgh, PA, USA, nihars@cs.cmu.edu
Conflict of Interest Disclosures
None reported.
Funding/Support
This work was supported in part by the US National Science Foundation Career award 1942124 and a Google Research Scholar Award, both of which support research on the fundamentals of learning from people with applications to peer review.
Role of the Funder/Sponsor
The sponsors did not play a direct role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.
Additional Information
The code is available at https://github.com/sjecmen/optimal_strategyproof_assignment.