Abstract
Development and Pilot Test of Risk of Bias Assessment Tool for Use in Peer Review
Brian S. Alper,1 Joanne Dehnbostel,1 Khalid Shahin,1 Amy Price,2 for the COVID-19 Knowledge Accelerator (COKA) Initiative
Objective
Peer review of clinical research should include risk of bias assessment, but it is often limited and not systematic. The Risk of Bias Assessment Tool (RoBAT) is an open web-based tool in which the user can select types of bias from a hierarchical list and record their assessment to help document a comprehensive risk of bias assessment for any scientific study. The RoBAT Usability Research Pilot Study was done to inform development of the tool and measures of its initial effectiveness, efficiency, and satisfaction.
Design
Participants were recruited through email distribution lists of COVID-19 Knowledge Accelerator, Guidelines International Network, Health Level Seven International, Healthcare Information For All, and International Society for Evidence-based Health Care (January 28-31, 2022). An online platform enabled participants to attest to meeting eligibility criteria (experience or education regarding risk of bias assessment and willingness to complete the study online), consent to participate, and complete usability evaluation reports for each assessment they attempted. Participants were encouraged to complete assessments at least 3 times (including 1 without RoBAT, 1 with RoBAT, and 1 with RoBAT after study-generated improvements). Surveys for each assessment included whether the task was completed (participant defined), time-on-task (participant reported as number of minutes), perceived ease of task (5-point scale), perceived ease of use of the tool (5-point scale), and open-ended questions for likes, dislikes, and suggested improvements. End-of-study surveys included the System Usability Scale1 and perceived usefulness for peer review support. Study enrollment closed after more than 5 participants used RoBAT, the number needed to detect 80% to 85% of issues for the initial discovery of usability problems.2,3
Results
A total of 18 participants were enrolled in the study and 10 completed 32 risk of bias assessment attempts. Task completion was achieved for 6 of 7 (86%) attempts without RoBAT, 13 of 17 (77%) attempts with the initial version of RoBAT, and 5 of 8 (63%) attempts with the revised version of RoBAT. Median (range) time on task was 34.5 (5-60) minutes without RoBAT, 30 (5-90) minutes with the initial version of RoBAT, and 20 (14-120) minutes with the revised version of RoBAT. The most common suggestions for improvements were to add instructions and to facilitate rapid selection of recognized terms. Data were too limited to establish a pattern for ease-of-use ratings from no to initial to revised RoBAT use. Five of 9 participants reported they would likely use RoBAT for peer review support.
Conclusions
The pilot study provided preliminary evidence of efficiency and satisfaction with RoBAT and demonstrated the feasibility of rapid online research development and implementation. Subsequent developments could test the tool’s usefulness for systematic reviewers and journal editors and integration with editorial systems.
References
1. Lewis JR. The System Usability Scale: past, present, and future. Int J Hum Comput Interact. 2018;34(7):577-590. doi:10.1080/10447318.2018.1455307
2. Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. In: CHI ’93: Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems. Association for Computing Machinery; 1993:206-213. doi:10.1145/169059.169166
3. Virzi RA. Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors. 1992;34(4):457-468. doi:10.1177/001872089203400407
1Scientific Knowledge Accelerator Foundation, Computable Publishing LLC, Ipswich, MA, USA, balper@computablepublishing.com; 2Anesthesia Informatics and Media Lab, Stanford University School of Medicine, Palo Alto, CA, USA
Conflict of Interest Disclosures
Brian S. Alper owns Computable Publishing LLC. Joanne Dehnbostel and Khalid Shahin are employees of Computable Publishing LLC. No other disclosures were reported.