Abstract
Experience With Select Crowd Review in Peer Review for The Thoracic and Cardiovascular Surgeon
Roman Gottardi,1 Peter Henning,2 Jessica Bogensberger,2 Markus K. Heinemann3
Objective
To evaluate the experience with a new peer review method,1 Select Crowd Review (SCR), for The Thoracic and Cardiovascular Surgeon. Upon submission, authors are given a choice to accept or decline SCR. A “crowd” was created in part from the existing reviewer pool and in part newly recruited. If authors agree to SCR, anonymized PDFs of manuscripts are made accessible to the crowd upon invitation per email via an online platform for 10 days. This intuitive platform enables the crowd to enter comments directly into the text; a formal structured review is not required. Reviewers give their comments anonymized. An SCR editor summarizes the annotations and gives a recommendation. Both the commented PDF and summary are sent back to the authors through the editor in chief. The aim is to achieve a rapid and broader, and thus fairer, review process.
Design
All manuscript submissions from introduction in July 2021 until January 15, 2022, were analyzed regarding acceptance and quality. Anonymized cardiac manuscripts were sent to a preselected crowd of 45 reviewers and entered regular double-blinded peer review at the same time. Efficiency and performance of the crowd’s reviews were compared with that of regular reviewers. For thoracic manuscripts, a crowd was not yet available during this pilot period.
Results
Of 162 total submitted manuscripts, 72 (44.4%) were selected for SCR; 84 were cardiac manuscripts, 39 (46.4%) of which were selected for SCR. Ten of those had to be rejected without any review, and 29 finally entered SCR. A first review process was completed for 24 manuscripts. For 3 manuscripts, the crowd did not respond. In all remaining 21 papers, the crowd’s recommendation concurred with that of the regular reviewers, leading to 8 rejections. Regular peer review took up to 5 weeks. Nine manuscripts underwent repeated SCR after revision. On average, 3 (range, 0-9) crowd members sent in reviews. In revisions, average response was worse, with mostly only 1 previous reviewer responding.
Conclusions
SCR encountered good acceptance by authors. Because the first experience showed absolutely concordant recommendations within 10 days compared with the slower traditional review, thoracic manuscripts have been included to gain more experience. If positive feedback continues, SCR may become an established method of peer review in selected journals. It is certainly helpful to achieve a fast first evaluation. Efficiency apparently must be increased for re-review of revisions.
Reference
1. List B. Crowd-based peer review can be good and fast. Nature. 2017;546(7656):9. doi:10.1038/546009a
1Mediclin Herzzentrum Lahr, Lahr, Germany; 2Thieme Medical Publishers, Stuttgart, Germany; 3Cardiovascular Surgery, Universitaetsmedizin Mainz, Mainz, Germany, heinemann@uni-mainz.de
Conflict of Interest Disclosures
Peter Henning and Jessica Bogensberger are employees of Thieme Medical Publishers and thus have an interest to have this new peer review method, developed by Thieme, evaluated for further use. Roman Gottardi is the SCR editor and Markus Heinemann is the editor in chief of The Thoracic and Cardiovascular Surgeon. They have no conflicts.