Abstract
A Pilot Study of Online Training of Patient Stakeholders Aimed at Improving Knowledge and Skills to Complete Peer Review of PCORI Draft Final Research Reports
Karen B. Eden,1 Ilya Ivlev,1 Amy Forester,2 Camber Hansen-Karr,1 Ed Reid,1 Lauren Saxton,1 Kelly Vander Ley,1 Mark Helfand1
Objective
Many patient, caregiver, and patient advocate reviewers for the Patient-Centered Outcomes Research Institute (PCORI) have not been formally trained in peer review. This pilot study was designed to develop new patient-oriented web-based PCORI peer review training and evaluate the effect of the new training on participants’ knowledge and self-efficacy to serve as a peer reviewer.
Design
In September 2016, we conducted a qualitative study to assess complexity of the task of reviewing a PCORI Draft Final Research Report (DFRR) and to identify training needs. We then conducted a pilot before-after study of early users of a PCORI patient peer review web-based training between February 2017 and May 2017. The training site, hosted by Sakai, Oregon Health & Science University’s learning management system, included videos on PCORI and the peer review process, a learning activity based on a DFRR annotated by editors, a writing activity, sample reviewer comments that were more helpful and less helpful, and online resources. Using site analytics within Sakai, access to the 6 training components was tracked. In this pilot study, we analyzed pretraining and posttraining knowledge (15 questions) and skills (6 questions) using McNemar tests; self-efficacy and attitude toward peer review were analyzed using Wilcoxon tests.
Results
Fourteen participants were interviewed about their experience reviewing a DFRR and 33 reviewers (14 patients, 18 patient advocates, and 1 caregiver) completed the pilot cross-sectional, before-after phase (33 of 52 [63%] completed pretraining and posttraining surveys and reported completing the training). Twelve of 14 interviewed participants reported having difficulties understanding from what point of view they needed to assess a technically written DFRR and reported a need for training in peer review. After the training was developed, 28 of 33 participants in the before-after study accessed 4 or more of the 6 training components. The percentage of correct answers, across participants, increased significantly after training for the knowledge questions (343 [71.5%] vs 441 [91.9%] of 480 answers; P < .001) (Table) and skills questions (143 [74.5%] vs 170 [88.5%] of 192 answers; P < .001). The number of questions answered as unsure significantly decreased for both knowledge questions (62 [12.9%] vs 6 [1.3%] of 480 questions; P < .001) and skills questions (19 [9.9%] vs 3 [1.6%] of 192 questions; P < .001). Importantly, the training increased confidence in patient stakeholders that they can complete a high-quality peer review (Z = −2.69; P = .007). All 33 participants reported that the training enhanced their knowledge and that they would recommend training to other patient or caregiver reviewers.
Conclusions
Completion of an online peer review training program shows promise in increasing knowledge and skills of reviewers. In the future, we plan to evaluate the dose of online training with the quality of actual peer reviews by patients and caregivers.
1Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, OR, USA, edenk@ohsu.edu; 2Teaching and Learning Center, Oregon Health & Science University, Portland, OR, USA
Conflict of Interest Disclosures:
None reported.
Funding/Support:
This study was conducted under a contract to provide peer review services for the PCORI DFRRs. Karen B. Eden, Amy Forester, Camber Hansen-Karr, Ed Reid, Lauren Saxton, Kelly Vander Ley, and Mark Helfand receive support from PCORI for developing the patient peer review training program and reviewing reports. Ilya Ivlev is a postdoctoral fellow supported by grant T15LM00708822 from the National Library of Medicine.
Role of the Funder/Sponsor:
PCORI was not involved in conducting this research study, but PCORI members were part of the team that prepared materials (videos, review form) for the patient and caregiver training. PCORI reviewed and approved the abstract for submission.
Acknowledgments:
We thank Kira Lesley for her timely help in recruiting patients, patient advocates, and caregivers.