Abstract
Trends in Peer Review Metrics at the Annals of African Surgery
James Kiilu,¹ Cecilia Munguti,¹ James Kigera,¹ Michael Mwachiro¹
Objective
The peer review process faces the challenges of recruiting reviewers and assessing the quality of the peer review process and manuscripts.1 This study evaluated trends in reviewer performance (R-scores), manuscript quality (M-scores), and their association with reviewer recommendations and editorial decisions at the Annals of African Surgery over 7 years.
Design
This retrospective study analyzed peer review reports at the Annals of African Surgery from 2017 to 2023, using data extracted from the journal’s manuscript management system. Variables included manuscript ID, submission year, revision cycles, reviewer ID, recommendations, R-scores, M-scores, time to review, and editorial decisions. R-scores assessed review timeliness and relevance, while M-scores evaluated the manuscript across dimensions such as interest, quality, and originality. Descriptive statistics summarized reviewer invitation responses, and cross-tabulations were performed to examine trends in R-scores and M-scores against submission year, reviewer recommendations, revision cycles, and editorial decisions. Associations between reviewer recommendations and editorial outcomes were assessed using χ² tests. Continuous data were tested for skewness, with analysis of variance applied to normally distributed data and Kruskal-Wallis tests to skewed data.
Results
For the period under review, the journal had a total of 1193 reviewers, with 370 (31%) coming from the international community. The number of reviewers invited per manuscript varied significantly across years, as determined by an independent-samples Kruskal-Wallis test (H2 = 18.46; df = 6; P = .005). The mean (SD) number of reviewers invited per manuscript was 6.35 (3.4) in 2017, peaked at 9.81 (9.6) in 2019, then declined to 5.12 (3.5) in 2021 before rising again to 7.02 (4.03) in 2023. There was a significant improvement in the reviewer acceptance rate, increasing from 29.9% in 2019 to 45.7% in 2023 (P < .001) (Table 25-0922). R-scores decreased significantly over the study period, from mean (SD) 2.68 (0.53) in 2017 to 2.29 (0.65) in 2023 (P < .001), while there was no significant change in M-scores. There was no statistically significant change in R-scores across different manuscript revision cycles, but M-scores showed significant improvement (from mean [SD] 2.60 [0.84] in the initial round of review to 2.17 [0.61] after 3 revisions, F = 2.413; P < .05), accompanied by a marked reduction in review turnaround time (from mean [SD] 11.07 [9.9] to 1.29 [1.38] days; P = .01). R-scores and M-scores varied significantly by editorial decision, with the highest R-scores observed for manuscripts receiving a major revision decision (mean = 2.71; P < .001) and the highest M-scores seen in rejected manuscripts (mean = 3.91; P < .001). One-way analysis of variance demonstrated statistically significant differences in mean R-scores awarded by the 15 editors (F14, 3986 = 17.77; P < .001). Reviewer recommendations were associated with editorial decisions (Pearson χ² = 288.80; df = 9; N = 1173; P < .001).
Conclusions
While reviewer acceptance rates improved significantly over time, there was no corresponding improvement in R-scores or M-scores. Future efforts should prioritize not only increasing reviewer participation but also strengthening the quality and consistency of reviewer feedback to more effectively support manuscript development.
Reference
1. Peterson CJ, Orticio C, Nugent K. The challenge of recruiting peer reviewers from one medical journal’s perspective. Proc Bayl Univ Med Cent. 2022;35(3):394-396. doi:10.1080/08998280.2022.2035189
1Annals of African Surgery, Nairobi, Kenya; james.kiilu@annalsofafricansurgery.com.
Conflict of Interest Disclosures
James Kigera is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.