Abstract
Content Analysis of Comments on bioRxiv and medRxiv Preprints
Clarissa França Dias Carneiro,1,2 Danielle Rayêe,3 Flávia Boos,4 Gabriel Costa,1 Kleber Neves,1 Mariana Boechat de Abreu,5 Pedro Batista Tan,1 Roberta Andrejew,6 Tiago Lubiana,7,8 Mario Malički,9 Olavo B. Amaral1
Objective
This study aimed to describe and explore the content of comments received by preprints posted in 2020 to bioRxiv and medRxiv. It extended a previous study1 that assessed comments on bioRxiv until 2019, considering the increased attention received by preprints after the beginning of the COVID-19 pandemic.2
Design
This was an observational qualitative study. The list of preprints posted to bioRxiv and medRxiv and the number of comments received by them were obtained using each platform’s application program interface. After excluding preprints with no comments and those with more than 20 comments (for feasibility and avoiding overrepresentation of individual preprints), eligible preprints were randomly sampled and their comments were assigned to 3 reviewers. Content analysis of comments was based on a structured form, with predefined categories based on previous research on preprint comments1 and on a scoping review of peer review.3 If 2 of 3 reviewers agreed, the classification was considered final; in case all 3 disagreed, a fourth reviewer made the final call. Neither manuscripts nor the comment appropriateness or validity in regard to the manuscript text were assessed.
Results
A total of 1037 preprints were assessed, with a total of 1921 comments (mean [SD], 1.8 [1.9] comments per preprint); 439 comments (22.8%) were replies to previous comments, leaving 1482 comments available for classification based on our prespecified categories (Table 19). A total of 14.6% (95% CI, 12.7%-16.5%) of reader comments were considered not to be about the content of the preprints (n = 192). Comments from preprint authors (n = 165) accounted for 11.1% (95% CI, 9.5%-12.7%) of the remaining comments, and mostly addressed publication status (n = 89); 38% (95% CI, 35.2%-40.9%) of reader (nonauthor) comments included compliments (n = 428), 61.7% (95% CI, 58.8%-64.5%) included criticisms, corrections, or suggestions (n = 694), and 34.9% (95% CI, 32.1%-37.7%) included questions (n = 393). Although compliments were mostly general (n = 279), criticisms, corrections, or suggestions largely addressed specific points, such as interpretation (n = 286), methodologic design (n = 267), materials/data collection methods (n = 238), and analyses (n = 228). Most questions were about information not provided in the preprint (n = 170) or about materials/data collection (n = 166).
Conclusions
As shown before,1 only a small percentage of preprints received comments on their respective preprint platforms, but this study found that the content of these comments showed similarities with the type of content typically expected from journal-elicited peer review as previously described.3 These results may help generate hypotheses to inform future research on preprints and peer review.
References
1. Malički M, Costello J, Alperin JP, Maggio LA. Analysis of single comments left for bioRxiv preprints till September 2019. Biochem Med (Zagreb). Published online April 15, 2021. doi:10.11613/BM.2021.020201
2. Fraser N, Brierley L, Dey G, et al. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLoS Biol. Published online April 2, 2021. doi:10.1371/journal.pbio.3000959
3. Glonti K, Cauchi D, Cobo E, Boutron I, Moher D, Hren D. A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. BMC Med. Published online June 20, 2019. doi:10.1186/s12916-019-1347-0
1Institute of Medical Biochemistry Leopoldo de Meis, Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil, clarissa.franca@bioqmed.ufrj.br, clarissa.franca-dias-carneiro@charite.de; 2BIH QUEST Center for Transforming Biomedical Research, Berlin Institute of Health, Berlin, Germany; 3Ophthalmology and Visual Sciences Department, Albert Einstein College of Medicine, New York, NY, USA; 4Programa de pós-graduação em Psicobiologia, Universidade Federal de São Paulo, São Paulo, Brazil; 5Carlos Chagas Filho Institute of Biophysics, Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil; 6Department of Biochemistry, Institute of Chemistry, Universidade de São Paulo, São Paulo, Brazil; 7Ronin Institute, virtual organization; 8Faculdade de Ciências Farmacêuticas, Universidade de São Paulo, São Paulo, Brazil; 9Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA
Conflict of Interest Disclosures
None reported.
Funding/Support
Clarissa França Dias Carneiro received a scholarship from the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq). Olavo B. Amaral received funding from FAPERJ (E-26/200.824/2021), CNPq (310813/2021-2), and the Serrapilheira Institute.
Role of the Funder/Sponsor
The funders had no direct role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.
Acknowledgment
The authors thank Richard Abdill for his contributions to data collection.
Additional Information
Mario Malički and Olavo B. Amaral are co–corresponding authors.