Abstract
Comparison of Content in Published and Unpublished Peer Review Reports
Elena Álvarez-García,1 Daniel Garcia-Costa,1 Flaminio Squazzoni,2 Mario Malički,3,4,5 Bahar Mehmani,6 Francisco Grimaldo1
Objective
While the publication of review reports increases the transparency of peer review, little is known about the effect it has on the content or quality of reports.1 The objective of our study was to assess differences in length, information content, and similarity of open vs unpublished reports.
Design
This was a cross-sectional study following STROBE guidelines for reporting2 that compared open with unpublished reports from 233 medical journals from Elsevier and Springer Nature, submitted from 2016 to 2021. Reports were obtained through confidential agreements with the publishers. Content of reports was compared using number of sentences, previously validated models for classification of 8 categories of content,3 information content score (mean cumulative distribution function values of each category’s sentence count, based on a zipfian distribution, with scores ranging from 0 to 1, where 1 indicates the highest informativeness), and information score dispersion measured by the Gini index (measure of similarity between reports for the same manuscript, where 0 indicates total similarity and 1 indicates total dissimilarity). Number of sentences, information score, and dispersion were compared using Mann-Whitney U tests. The association of information score and Gini index with peer review type, journal quartile of impact factor, and reviewer gender, seniority, and region were explored using generalized linear models. To adjust for multiple comparisons, we considered P < .001 as statistically significant.
Results
Compared with unpublished reports (n = 117,250), open peer review reports (n = 40,844) were longer (median [IQR] length, 22 [12-37] sentences vs 17 [10-28] sentences; P < .001) than unpublished reports and had a higher informative content (median [IQR] score, 0.52 [0.35-0.70] vs 0.46 [0.30-0.65]; P < .001), with the largest difference found in the number of suggestion and solution sentences (Figure 25-1081). Women’s reports had a higher information score than men’s reports (difference, 6.3%), and reviewers from non-Western institutions had lower scores than those from Western institutions (difference, −6.0%). Open peer review reports were also more similar to each other (median [IQR] Gini index, 0.19 [0.09-0.33] vs 0.22 [0.11-0.35]; P < .001).
Conclusions
Our study showed that open peer review reports were longer than traditional unpublished reports, with the greatest differences found in the number of suggestion and solution sentences. These results suggest that increasing the transparency of peer review could lead to more detailed reports that focus on manuscript improvement.
A limitation of our study is that we did not have access to published manuscripts and were unable to determine the impact of initial quality of manuscripts beyond journal ranking, nor the impact peer review reports had on manuscript improvement.
References
1. Ross-Hellauer T, Horbach SP. Additional experiments required: a scoping review of recent evidence on key aspects of open peer review. Res Eva. Published online February 8, 2024. doi:10.1093/reseval/rvae004
2. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495-1499. doi:10.1016/j.ijsu.2014.07.013
3. Severin A, Strinzel M, Egger M, et al. Relationship between journal impact factor and the thoroughness and helpfulness of peer reviews. PLoS Biol. 2023;21(8):e3002238. doi:10.1371/journal.pbio.3002238
1Department of Computer Science, University of Valencia, Valencia, Spain; 2Department of Social and Political Sciences, University of Milan, Milan, Italy; 3Stanford Program on Research Rigor and Reproducibility, Stanford University, Stanford, CA, US, mmalicki@ stanford.edu; 4Department of Epidemiology and Population Health, Stanford University, Stanford, CA, US; 5Meta-Research Innovation Center at Stanford, Stanford University, Stanford, CA, US; 6STM Journals, Elsevier, Amsterdam, the Netherlands.
Conflict of Interest Disclosures
Mario Malički was an Editor in Chief of Research Integrity and Peer Review, which is published by Springer Nature, provider of the part of the dataset. Bahar Mehmani is an employee of Elsevier, provider of the other part of the dataset. Bahar Mehmani is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.
Funding/Support
Elena Álvarez-García, Daniel GarciaCosta, and Francisco Grimaldo have been partially supported by the Regional Ministry of Education, Culture, Universities and Employment of the Generalitat Valenciana under project CIAICO/2022/154. Flaminio Squazzoni was supported by a grant from the Progetti di Rilevante Interesse Nazionale—Italian Ministry of University and Research (grant 202297CKET_00 “ALGOLIT”).
Role of the Funder/Sponsor
The funder was not involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.
Acknowledgment
We gratefully acknowledge the support of Springer Nature, which provided the open peer review reports for our study, specifically Suzuki Limbu, Attilia Czikmantori, and their production team. We also thank all reviewers of Springer Nature journals that participated in signing or allowing their review reports to be published and all journals and editors facilitating these processes. We acknowledge the support on data extraction from the IT staff of Elsevier, specifically Ramsundhar Baskaravelu and his team. This work uses Scopus data provided by Elsevier through the Peer Review Workbench. We also thank Dave Santucci from the Elsevier Scopus API team and Kristy James from the International Center for the Study of Research for their support on data enrichment about authors and reviewers. We thank Josep Monclús for his help on data extraction and preparation during the initial phase of this study.