Abstract

Association Between Peer Reviewers’ Priority Ratings of Impact of Research Manuscripts With Citations and Altmetric Scores of Subsequently Published Articles in the Journal of Medical Internet Research

Gunther Eysenbach1

Objective

Peer-reviewed journals ask reviewers to rate the perceived impact or priority of a manuscript. Previous research has suggested an association between reviewer priority scores and citations.1 Altmetrics (alternative metrics) provide an alternative view on social impact (ie, uptake on social media, in policy documents, or by news articles2); however, their association with reviewer scores has not been explored.3 It is unclear whether reviewer ratings more closely reflect scientific impact (ie, citations) or social impact (altmetric scores), and which metric is more valid to reflect reviewers’ priority ratings.

Design

This was a longitudinal bibliometric cohort study that followed 451 original research articles published in 2018 in the Journal of Medical Internet Research, a health services research and medical informatics journal (impact factor, >5). This journal asks peer reviewers to rate the priority (defined as potential impact) of a manuscript on an ordinal rating scale with possible scores of 1, 2, 5, and 10 (highest priority). Manuscripts are typically reviewed by 2 reviewers. The mean priority score of all reviewers for a manuscript in the first review round constitutes the Manuscript Average Priority Score (MAPS). For this analysis, manuscripts were categorized into 4 quartiles (Qs), with the groups labeled as Q4 (MAPS score, ≤3) to Q1 (MAPS score, >5). The dependent variables, citations, and altmetric scores were obtained from the Dimensions database in February 2022; manuscripts and published articles were similarly stratified into quartiles, with the citation (or altmetrics) quartile Q1 containing the group of articles with the highest citation count (or altmetric score). The association between independent variables (MAPS scores) and citation or altmetric scores was measured using χ² tests for 4 × 4 contingency tables for the quartiles and using Spearman rank correlation between MAPS score ranks and citation or altmetric rank, respectively.

Results

The MAPS scores for 451 published articles ranged from 1.5 to 10; citations, from 0 to 253; and altmetric scores, from 1 to 849. Although both mean and median citations as well as altmetric scores were higher in the higher MAPS quartiles (Table 46), the results of χ² tests were not statistically significant for citations (P = .46) but were statistically significant for altmetric scores (P = .03). The Spearman rank correlation between citation ranks and MAPS score ranks was statistically significant but weak (ρ = .0955; r2 = .009; P = .03). In contrast, altmetric score ranks had a stronger correlation with MAPS score ranks (ρ = .1313; r2 = .017; P = .002).

Conclusions

This longitudinal bibliometric cohort study found that in the Journal of Medical Internet Research, a journal whose subject matter lends itself to the type of attention measured by altmetrics, altmetric scores seemed to be better correlated than citations with a manuscript’s potential impact as assessed by reviewers. Peer reviewers may interpret priority and impact in terms of social impact, rather than citations, raising further questions about the appropriateness of citation-based metrics to measure impact as understood by reviewers.

References

1. Opthof T, Coronel R, Janse MJ. The significance of the peer review process against the background of bias: priority ratings of reviewers and editors and the prediction of citation, the role of geographical bias. Cardiovasc Res. 2002;56(3):339-346. doi:10.1016/S0008-6363(02)00712-5

2. Eysenbach G. Can tweets predict citations? metrics of social impact based on twitter and correlation with traditional metrics of scientific impact. J Med Internet Res. 2011;13(4):e123. doi:10.2196/jmir.2012

3. Araujo AC, Vanin AA, Nascimento DP, et al. What are the variables associated with altmetric scores? Syst Rev. 2021;10:193. doi:10.1186/s13643-021-01735-0

1JMIR Publications, Toronto, ON, Canada, geysenba@gmail.com

Conflict of Interest Disclosures

Gunther Eysenbach reported equity in JMIR Publications.

Poster