Abstract
Quality of Patient Reviewer Comments and Association With Author and Editor Responses
Melecia Miller,1 Vera Nezgovorova,2 Ilana Kersh,3 Mohamed I. Elsaid,4 Marina Broitman2
Objective
This study’s goal was to develop a way to assess patient review quality based on author and editor reactions to reviews. We used patient reviews of draft final research reports (DFRRs) of Patient Centered Outcomes Research Institute (PCORI)–funded research that were peer reviewed by clinical, statistical, and patient reviewers.1 Review quality ratings were based on prior research on PCORI peer review and other research on patient reviewers.2,3 We hypothesized that specific, substantive changes to the report and patient-focused comments would be associated with a favorable mention of the patient review by the editor as well as a substantive response from the author.
Design
Pilot testing on 13 patient reviews for 13 clinical research DFRRs submitted from 2021 to 2022 was completed in 2025. We conducted a content analysis of patient reviewer comments and developed a coding scheme based on prior research. Coding was performed at the sentence level and included 2 categories: (1) specificity and constructiveness of the comment and (2) patient focus of the comment (Table 25-1110). Each review was coded independently by 2 raters each for category 1 and category 2, with a final Cohen κ of 0.83 and 0.66, respectively, indicating moderate to substantial interrater agreement. Coding discrepancies between reviewers received the most conservative (ie, lowest value) code. Raters assessed whether the editor mentioned the patient review at least once in their review summary and either encouraged (code 1) or discouraged (code 2) an author response vs no mention (code 0). Author responses to reviewer comments were coded by paragraph indicating whether authors responded directly to patient-reviewer comments (code 1) or made changes to the report because of the comments (code 2) vs neither (code 0). We used hierarchical modeling to calculate the odds of editors encouraging authors to respond (code 1 vs codes 0 or 2) and mixed-effects logistic regression to calculate the odds of authors responding substantively (codes 1 or 2 vs code 0) to patient reviews with more specific, constructive comments.
Results
Table 25-1110 shows counts for associations between category 1 and category 2. More specific and constructive comments and patient-focused comments were not associated with higher odds of the editor mentioning the patient review (N = 13; odds ratio [OR], 0.36; 95% CI, 0.28-0.45 and OR, 0.65; 95% CI 0.54-0.78, respectively). Patient review paragraphs with more than 50% specific and constructive comments were associated with higher odds of authors responding substantively, compared with other comment types (N = 147; OR, 4.00; 95% CI, 1.67-9.59; P = .002).
Conclusions
This pilot study shows the feasibility of establishing codes rating the quality of patient reviews and determining whether higher quality patient reviews are associated with mentions by the editor and author responses. The results are limited because of the sample size and the uniqueness of DFRRs.
References
1. Broitman M, Sox HC, Slutsky J. A model for public access to trustworthy and comprehensive reporting of research. JAMA. 2019;321(15):1453-1454. doi:10.1001/jama.2019.2807
2. Ghazaryan E, Broitman M, Sox H. Differences in the style and quantity of reviewer comments in structured vs unstructured peer review forms. Poster presented at: Ninth International Congress on Peer Review and Scientific Publication; September 10, 2022; Chicago, IL.
3. Huml AM, Albert JM, Beltran JM, et al. Community members as reviewers of medical journal manuscripts: a randomized controlled trial. J Gen Intern Med. 2022;38(6):1393-401. doi:10.1007/s11606-022-07802-z
1KnowledgeWorks Global, Ltd., 410 MN-25, Brainerd, MN, US; mmiller@originreview.org; 2Patient-Centered Outcomes Research Institute, Washington, DC, US; 3University of Pittsburgh, Pittsburgh, PA, US; 4The Ohio State University College of Medicine, Columbus, Ohio, US.
Conflict of Interest Disclosures
None reported.