Abstract
Differences in the Style and Quantity of Reviewer Comments in Structured vs Unstructured Peer Review Forms
Emma Ghazaryan,1 Marina Broitman,2 Harold Sox2
Objective
This study compared 2 critique formats used for subject matter expert (SME) reviews for Patient-Centered Outcomes Research Institute (PCORI) draft final research reports (DFRRs).1 In November 2018, PCORI changed SME critique formats from 13-item structured forms to unstructured forms after DFRR authors commented about the burden and repetitiveness of critiques. Existing review quality tools were examined2,3 but focused more on scientific review quality than comment type.
Design
Subject matter expert reviews from all clinical research reports that completed peer review between October 2017 and September 2019 (13 months before and 11 months after the review form change) were compared to see whether structured and unstructured reviews had the same number and types of comments. In September 2019, the abstract authors defined comment categories for SME reviews: (1) general positive or complimentary comments, (2) critical comments leading to a report change, (3) critical comments not leading to a change, (4) grammatical errors, (5) comments repeating an earlier statement by the same reviewer, and (6) neutral factual comments. The first author (E.G.) developed the categories based on previous tools2 and iteratively revised the categories on the basis of discussions and coding comparisons with the second author (M.B.). The responses of DFRR authors to the SME review comments were used to confirm types of comments. The study outcome was the number of comments in each category as rated by the first author. Interrater reliability was determined by category agreement between the first author and a colleague unfamiliar with this project on 124 reviewer comments, resulting in a kappa of 0.69. The percentage of each comment category and the mean (SD) number of comments per SME review were calculated. Given the small sample, analyses were descriptive.
Results
The sample included 49 DFRRs (30 before [61%] and 19 after [39%] the review form change), with a total of 99 SME reviews (2.1 SME reviews per DFRR); 61 (62%) used the structured form and 38 (38%) used the unstructured form. Including all comment categories, the mean (SD) number of comments per review was 39.4 (17.7) with the structured form and 17.7 (9.9) with the open-ended form (Table 70). With the unstructured form, positive or complimentary comments were a larger percentage of all comments (36.7% vs 30.4%), whereas the percentage of repetitive comments was lower (2.2% vs 13.4%) compared with the structured form. With both forms, approximately 40% of comments were critical comments that led to a change in the DFRR.
Conclusions
Structured review forms provided more reviewer comments than unstructured forms but included more repetitive comments and a smaller proportion of positive comments. The study sample was too small to address the association of these differences with the quality or speed of peer review.
References
1. Broitman M, Sox HC, Slutsky J. A model for public access to trustworthy and comprehensive reporting of research. JAMA. 2019;321(15):1453-1454. doi:10.1001/jama.2019.2807
2. van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999;52(7):625-629. doi:10.1016/s0895-4356(99)00047-5
3. Superchi C, González JA, Solà I, Cobo E, Hren D, Boutron I. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC Med Res Methodol. 2019;19(1):48. doi:10.1186/s12874-019-0688-x
1World Health Organization, Yereven, Armenia; 2Patient-Centered Outcomes Research Institute, Washington, DC, USA, mbroitman@pcori.org
Conflict of Interest Disclosures
Emma Ghazaryan’s spouse is employed by the National Institutes of Health. Marina Broitman’s spouse is employed by the National Institutes of Health. No other disclosures were reported.
Funding/Support
This work was funded by the Patient-Centered Outcomes Research Institute (PCORI).
Role of the Funder/Sponsor
PCORI provided review and approval of the abstract for submission to the Peer Review Congress.
Disclaimer
This work was completed by Emma Ghazaryan when she was employed as an intern at PCORI. Marina Broitman and Harold Sox are PCORI employees. All statements, findings, and conclusions in this publication are solely those of the authors and do not necessarily represent the views of PCORI or its board of governors.