Abstract
Use of a 3-Round Modified Delphi Process to Support Rapid Peer Review
Sean Hays,1 Tyler Carneal,1 Christopher Kirman1
Objective
A pilot study was conducted in which a community of SciPinion science experts (>32,000 registered users) and the SciPinion web platform performed rapid peer review of a study.
Design
Review design elements were included to avoid sources of bias in key components of the peer review (reviewer participation, selection, engagement, and reporting). A review announcement, released in January 2025, included the general topic and expertise needs (eg, developmental neurotoxicity) but did not identify the specific manuscript. Within 24 hours of the announcement, 152 scientists applied and 5 reviewers were selected automatically by the web platform based on expertise metrics (years of experience, publications, counts of key words relevant to the manuscript). The review panel consisted of qualified scientists (mean [SD] years of experience, 37 [14] years; range, 21-52 years; mean [SD] publication count, 416 [124]; range, 216-550) across 4 countries. Reviewers were automatically verified via a community-driven reputation system based on past interactions with the web platform. To improve robustness of the process, the review was structured as a 3-round modified Delphi format. A $250 honorarium was offered to reviewers to complete their work under an accelerated schedule. Reviewers were masked to each other. In round 1, reviewers worked independently to review the manuscript and answer structured questions that included confidence ratings (0-10 scale, with higher number indicating higher confidence) for components of the manuscript with explanations and confidence in relying on the reported study for making policy decisions by health agencies. In round 2, reviewers worked deliberatively in reviewing and commenting on each other’s answers (13 comments submitted). In round 3, the reviewers returned to their round 1 responses, revised as needed after deliberation, and submitted final responses.
Results
All 3 rounds of the review were completed within 1 week of panel selection, and honoraria were initiated within 24 hours of completion. A comparison of the panel’s independent input from round 1 and their deliberative input in round 3 yielded a decrease in the confidence score of the manuscript (mean [SD] confidence score of 4.2 [2.5] after round 1 was 3.8 [2.2] after round 3) and an improvement in consensus across reviewers (based on Tastle and Wierman1), from 0.71 after round 1 to 0.77 after round 3) (Figure 25-1065). Confidence in using the study to support decision-making by health agencies was considered low by the review panel (mean [SD] score, 3.0 [1.2]).
Conclusions
This pilot study found that rapid (ie, within 1 week) peer reviews can be performed with a process that includes multiple rounds of engagement and collaborative sharing of reviewer ratings and comments in a scalable manner via automation of multiple steps in the process, including ranking of candidate reviewers, contracting (agreement defining scope of work and compensation), scheduling of rounds, messaging, and payments.
Reference
1. Tastle WJ, Wierman MJ. Consensus and dissention: a measure of ordinal dispersion. Int J Approx Reason. 2007;45(3):531-545. doi:10.1016/j.ijar.2006.06.024
1SciPinion, Bozeman, MT, ckirman@scipinion.com.
Conflict of Interest Disclosures
None reported.
Funding/Support
This work was self-funded by SciPinion.