Abstract
A Novel Open Peer Review Format for an Emergency Medicine Blog
Scott Kobner,1,2 Derek Sifford,2,3 Michelle Lin2,4
Objective
The medical education blog Academic Life in Emergency Medicine (ALiEM) has developed a new, open, inline peer review publication format that presents reviewer commentary within the body of the content. We hypothesized that the proximity and interactive nature of inline text will increase webpage engagement and investigated the association of this format with reader behavior.
Design
We wrote 2 ALiEM blog posts for the purpose of this study. Each post was reviewed by a pair of expert peer reviewers to produce 3 publication versions: (1) control post with no expert peer review; (2) traditional post with expert peer review appended to the end of the post content; and (3) experimental post, which used inline text throughout the content. Website visitors were randomized to view either the control, traditional, or experimental version using a custom Google Analytics Content Experiment. Google Analytics, paired with a customized JavaScript activity tracker, and CrazyEgg, an industry standard web visualization suite, captured user demographic information, active users every 5 seconds, bounce rate, user click activity, and user scroll activity over a 1-month time period. We compared measures using the χ² test.
Results
Data collection is in progress, and we report the first blog post’s preliminary analytic data. During the period from June 4 to 9, 2017, 502 views were captured across the 3 versions of the post (196 views of the control post, 149 views of the traditional post, and 157 views of the experimental post). Bounce rates were nearly identical across groups (73.6%-73.9%; P > .99). The time users spent viewing each blog post, measured as active users present over increments of 5 seconds, was not statistically significantly different (median time of 50 seconds [interquartile range, 20-90 seconds] viewing the control post, 50 seconds [interquartile range, 20-95 seconds] viewing the traditional post, and 45 seconds [interquartile range, 20-99 seconds] viewing the experimental post; P = .46). There were no apparent differences in user scrolling data. However, pages featuring inline peer review comments had a statistically significantly greater proportion of clicks on summative peer review content than posts with traditional commentary (15.21% experimental vs 8.92% traditional; P < .03).
Conclusions
The inline peer review publication system does not appear to alter reading times, bounce rates, or scrolling activity. Readers did, however, click to open peer review content more when viewing the inline version compared with the traditional post. The preliminary data suggest that expert inline commentaries might increase the value of scientific content published online by increasing the visibility of published peer review content for readers. Future studies should examine the association of transparent, inline expert peer reviews with reader cognitive load and learning.
1Department of Emergency Medicine, Los Angeles County and University of Southern California Medical Center, Los Angeles, CA, USA, scottkobner@gmail.com; 2Academic Life in Emergency Medicine, San Francisco, CA, USA; 3Wayne State University, Detroit, MI, USA; 4University of California, San Francisco at Zuckerberg San Francisco General, San Francisco, CA, USA
Conflict of Interest Disclosures:
All authors are members of the not-for-profit organization Academic Life in Emergency Medicine, LLC. Dr Kobner is the New Submissions Editor, Mr Sifford is the Chief Technology Officer, and Dr Lin is the Editor in Chief and Chief Executive Officer. All are paid contractors for the organization. They are compensated for their efforts in maintaining the blog and other digital innovations. No other disclosures were reported.