Abstract
Concordance Between Peer Reviewers’ Recommendations and Editorial Decision-Making at The Journal of Pediatrics
Raye-Ann deRegnier,1,2 Kevin Jewett,1,3 Meghan McDevitt,4 Denise M. Goodman1,2
Objective
To examine concordance between peer reviewers’ recommendations and editorial decisions for a general pediatric journal.
Design
This was a cross-sectional observational study of a convenience sample of peer-review recommendations for articles submitted to The Journal of Pediatrics reported using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline. After submission, an editor made a preliminary assessment and invited peer reviewers. Responding reviewers were asked to provide a recommendation for acceptance, revision, or rejection. When recommending acceptance or revision, reviewers were asked to select a priority score for publication on a 4-level Likert scale (low, medium low, medium high, high). After assessing the reviews, editors decided either to move the article forward (accept or invite a revision) or to reject. Manuscripts were included in the study if the original submission was initially reviewed by at least 2 invited reviewers. The primary outcome of the study was the final disposition (accept or reject). Reviewer recommendations and Likert scores in response to the original submission were compared with final dispositions using univariable and multivariable regression. For the multivariable analysis, the highest reviewer priority score was dichotomized as high/medium high vs. medium low/low. Differences between priority scores were dichotomized as matched or differing by 1 Likert level vs differing by 2 or 3 Likert levels.
Results
The sample included 1016 articles sent for review between June 16, 2020, and October 27, 2021, with at least 2 completed reviews in response to the original submission. At least 1 reviewer recommended rejection for 393 of 1016 manuscripts (38.7%). When reviewers agreed to reject, all manuscripts were rejected (65 of 65); however, when reviewers disagreed on rejection, 36 of 328 manuscripts were accepted for publication (11%; P = .005) and 1 is pending to date. Among articles with a reviewer recommendation for rejection and at least 1 reviewer priority score (n=294), high or medium high priority scores were associated with a final decision to accept the article (odds ratio [OR],11.37; 95% CI, 3.94-32.85; P < .001). Reviewers agreed to move the article forward for 623 of 1016 manuscripts (61.3%) with 395 ultimately accepted for publication. Of these 623, 494 (79.3%) had at least 2 reviewer priority scores and a final disposition. The highest reviewer priority score was significantly associated with the acceptance rate (Table 34). In multivariable regression, higher priority score category (OR, 12.59; 95% CI, 6.84-23.14) and level of reviewer agreement on priority scores (OR, 2.04; 95% CI, 1.15-3.6) were both significantly associated with article acceptance rates.
Conclusions
Both reviewer recommendations on acceptance or rejection and priority scores for publication appeared to be influential in editorial decision-making. However, editors were frequently required to arbitrate mixed reviewer recommendations and make independent decisions.
1The Journal of Pediatrics, r-deregnier@northwestern.edu; 2Ann & Robert H. Lurie Children’s Hospital of Chicago; 3Cincinnati Children’s Hospital; 4Patient-Centered Outcomes Research Institute
Conflict of Interest Disclosures
None reported.