Abstract
Monetary Incentives for Peer Review at a Medical Journal: A Quasi-Randomized Experimental Study
Christopher S. Cotton,1 Abid Alam,1 Sophie Tosta,2 Timothy G. Buchman,3 David M. Maslove4,5,6
Objective
Peer review is a cornerstone of academic publishing, providing a bulwark against the dissemination of flawed or unethical research. Historically, peer review has been carried out by experts on a volunteer basis. Financial compensation for peer reviewers has been discussed but with little empirical evidence to inform the debate.1 A single experimental study2 has examined this question, finding that for peer review at an economics journal, offering a $100 gift card to reviewers reduced median review time by 8 days. No experimental study to our knowledge has looked at the effect of paying for peer review at a medical journal. Our objective was to determine the effect of a cash incentive on the rate of fulfillment of peer review requests at a medical journal.
Design
We conducted a quasi-randomized experimental study at Critical Care Medicine, a specialty medical journal. We divided the experimental period (September 2023 to March 2024) into alternating 2-week blocks, during which all peer review requests sent out to prospective reviewers included either an offer of $250 to complete the peer review assignment (incentive blocks) or no offer (control blocks). Only original research reports were included in the study, with the monetary incentive applying only to the initial review. The primary outcome was the rate of invitation –to completed review conversion, defined as the ratio of reviews submitted divided by the number of reviewer invitations sent out. Review quality was a secondary outcome.
Results
During the study, 131 manuscripts went out for review, with a median (IQR) of 5 (2-8) reviewer invitations per manuscript. Of 715 reviewer invitations, 414 (58%) included an incentive offer. There was no difference in the proportion of invitations that were accepted (control vs incentive, 144 of 301 [48%] vs 218 of 414 [53%]; P = .20), but a difference was seen in the primary outcome, with reviewers in the intervention group having a greater odds of completing the review compared with those in the control group (control vs incentive, 127 of 301 [42%] vs 206 of 414 [50%]; odds ratio, 1.36; 95% CI. 1.01-1.84). A Cox proportional hazards model showed that incentivized invitations were fulfilled slightly faster (median [IQR] time, 11 [6-14] days vs 12 [7-14] days; hazard ratio, 1.30; 95% CI, 1.04-1.62). There was no difference in the quality of reviewer reports, as adjudicated by the handling editors (control vs incentive, median [IQR] score on a 100-point scale, 75 [70-85] vs 75 [66-90]).
Conclusions
Our findings suggest that cash payments can incentivize peer review at a medical journal, potentially providing a method to help journals expedite peer review in situations like public health emergencies. Important policy questions remain around cost-benefit considerations; extrapolating from the rate of reviews required during the study period, we estimate that offering $250 for each review would cost Critical Care Medicine nearly $150,000 annually. Other considerations include sources of funding for paid reviews, the effect of nonmonetary incentives, and the potential unintended consequences of paid reviews.
References
1. Cheah PY, Piasecki J. Should peer reviewers be paid to review academic papers? Lancet. 2022;399(10335):P1601. doi: 10.1016/S0140-6736(21)02804-X
2. Chetty R, Saez E, Sándor L. What policies increase prosocial behavior? an experiment with referees at the Journal of Public Economics. J Econ Perspect. 2104;28(3):169-188. doi: 10.1257/jep.28.3.169
1Department of Economics, Queen’s University, Kingston, ON, Canada; 2Society of Critical Care Medicine, Mount Prospect, IL, US; 3Emory Critical Care Center, Emory University School of Medicine, Atlanta, GA, US; 4Department of Critical Care Medicine, Queen’s University, Kingston, ON, Canada, david.maslove@queensu.ca; 5Department of Medicine, Queen’s University, Kingston, ON, Canada, 6Kingston Health Sciences Centre, Kingston, ON, Canada.
Conflict of Interest Disclosures
David M. Maslove was an associate editor at Critical Care Medicine at the time the study was conducted. Timothy G. Buchman was the editor-in-chief at Critical Care Medicine at the time the study was conducted. Sophie Tosta is the managing editor of Critical Care Medicine.
Funding/Support
This study was supported by a grant from the Government of Canada’s New Frontiers in Research Fund (grant No. NFRFR-2021-00335). David M. Maslove is supported by a New Clinician Scientist Award from the Southeastern Ontario Academic Medical Association. Christopher S. Cotton receives research funding as the Jarislowsky-Deutsch Chair in Economic & Financial Policy at Queen’s University.
Role of Funder/Sponsor
No funder had a role in the design, analysis, or interpretation of the study.
Acknowledgment
The authors wish to thank the associate editors and senior editors at Critical Care Medicine, as well as the editorial assistants at the journal, and the staff at the Society of Critical Care Medicine for facilitating the study.