Abstract

Linguistic Features in Peer Reviewer Reports: How Peer Reviewers Communicate Their Recommendations

Ketevan Glonti,1,2 Darko Hren,1 Simon Carter,3 Sara Schroter4

Objective

Many biomedical journals ask peer reviewers to provide a recommendation on the manuscript under review: whether to accept with no revision; minor revision; major revision, or to reject the manuscript. Some editors use these reviewer recommendations to help their editorial decisions. The content and linguistic features of peer reviewer reports may vary depending on the reviewer’s recommendation. Our aim was to identify trends in the style of language employed by peer reviewers when providing a recommendation.

Design

This is a retrospective analysis of peer reviewer reports collected during a previously reported single-blind randomized controlled trial carried out by The BMJ. That trial explored the effects of providing training to reviewers on the quality of peer reviewer reports of 3 manuscripts. For this study, we analyzed a random sample of 440 of 1372 reviewer reports in the control arm of the study. Of these, 330 were recommended for rejection and 110 were recommended for acceptance (8 were recommended for acceptance with no revision, 38 for acceptance with minor revision, 42 for acceptance with major revision, and 22 other recommendations. Peer reviewer reports were analyzed using automatic textual analysis software Linguistic Inquiry and Word Count (version 1.3.1, LIWC2015) to capture the rhetorical strategies (eg, analytic language) deployed in the text in 4 language variables: analytic, clout, authentic, tone. These LIWC scores for individual reviews were imported into an SPSS database (version 24, IBM) where descriptive statistics (means, standard deviations, and 95% CIs for means) were calculated and statistical tests performed using independent sample t tests.

Results

The Table summarizes key differences in language strategies used in peer reviewer reports that recommend acceptance or rejection of the manuscript being reviewed. There was no difference in reviewer reports recommending rejection vs acceptance in their analytic nature. Reviewer reports recommending rejection were significantly more ”authentic” (ie, had more frequent use of terms that indicate honesty and personal disclosure) and scored lower in terms of ”tone” (ie, emotional tone, where low scores indicate regret or hostility). Reviewer reports recommending rejection also scored lower in terms of “clout” (for which, high scores suggest that a review emerges from a perspective of high expertise and confidence on the part of the reviewer, and low scores reflect a more tentative or humble language style).

Conclusions

Many journals in the biomedical field request that peer reviewers make a recommendation on the fate of a submitted manuscript. Greater awareness and understanding of the type of feedback and linguistic features employed by peer reviewers to communicate their recommendation may enable editors to develop strategies for making better informed editorial decisions.

1Department of Psychology, School of Humanities and Social Sciences, University of Split, Split, Croatia, kglonti@unist.hr, 2Université Paris Descartes, Sorbonne Paris Cité, Paris, France, 3Department of Sociology, The Open University, Milton Keynes, UK, 4The BMJ, London, UK

Conflict of Interest Disclosures:

Ketevan Glonti and Darko Hren were supported by a grant by the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 676207. Sara Schroter is an employee of The BMJ and regularly undertakes research into the publishing process. Simon Carter reported no conflicts of interest.