Abstract
Manuscript Characteristics Associated With Editorial Review and Peer Review Outcomes at Science and Science Advances
Nicholas LaBerge,1 Sam Zhang,1,2,3 Quinten McElhiney,1 Daniel B. Larremore,1,2 Aaron Clauset1,2
Objective
To make publicly available a deidentified dataset of manuscript submissions and associated editorial metadata at Science and Science Advances (2 elite multidisciplinary journals) and quantify the manuscript and author characteristics associated with outcomes over the 2 stages of evaluation at these highly selective journals: (1) editorial review, when journal editors screen and select a smaller set of submissions for detailed consideration, and (2) peer review, when editors recruit outside experts to evaluate manuscripts. Peer review at elite scientific journals is a high-stakes process whose outcomes have a broad influence in science and society. However, the need to maintain peer review’s confidentiality has limited the range of data available for scientific study. This lack of peer review data makes it difficult to assess how well elite journals achieve the scientific community’s ideals or to improve and test theories to improve their evaluation processes.
Design
We introduced and described a manuscript-level dataset of 110,303 deidentified evaluations of manuscripts submitted to Science and Science Advances over a 5-year period (2015-2019), each with a standard set of author and manuscript characteristics, and we conducted a series of logistic regression analyses to quantify the correlates of success in the initial editorial review stage (desk rejections) and the subsequent peer review stage at both journals.
Results
Each manuscript record includes author, editor, and manuscript characteristics, including topic, team size, institutional prestige, gender, geographic region, evaluation scores, and editor decisions; personally identifiable and reidentifiable information, including the text of the reviews, is excluded. Our statistical analyses revealed strong associations with institutional prestige, team size, manuscript topic, and country, which are primarily attributable (via a mediation analysis) to the influence of the editor, even as the tenor of advice from outside experts correlates strongly with the final editorial decision (Figure 25-0972). Corresponding authors who are men appear to have a small but significant advantage with editors and reviewers at Science, while authors based in China have a significant disadvantage.
Conclusions
This deidentified dataset will support further investigation by peer review researchers into the 2-stage evaluation process of these 2 elite journals, and our results will help quantify the complex interactions among editors, reviewers, manuscript characteristics, and author characteristics at elite journals.
1University of Colorado Boulder, Boulder, CO, US, aaron.clauset@ colorado.edu; 2Santa Fe Institute, Santa Fe, NM, US; 3University of Vermont, Burlington, VT, US.
Conflict of Interest Disclosures
Aaron Clauset is a Deputy Editor at Science Advances; all other authors declare no conflicts of interest.
Funding/Support
This work was supported by Air Force Office of Scientific Research Award FA9550-19-1-0329 (to Nicholas LaBerge, Sam Zhang, Daniel B. Larremore, and Aaron Clauset), National Science Foundation (NSF) SBE Award 2219609 (to Nicholas LaBerge, Sam Zhang, Daniel B. Larremore, and Aaron Clauset), NSF Graduate Research Fellowship Award DGE 2040434 (to Sam Zhang), and NSF Alan T. Waterman Award SMA-2226343 (to Daniel B. Larremore).
Role of Funder/Sponsor
The funders had no role in this research.