Abstract
Assessment of a Standardized Tool to Identify Deceptive Journals
Kathleen Berryman,1 Sheree Crosby,1 Lacey Earle,1 Lucas Toutloff1
Objective
Cabell’s provides academic journal analytics to the scientific research community. Our objective is to develop an unbiased, transparent, and effective tool for identifying deceptive academic journals by analyzing specific behavioral indicators.
Design
For this study, we defined “deceptive” as intentional misrepresentation of facts or failure to provide implied services. Our first step was to identify behaviors common to journals recognized as deceptive. We examined 261 randomly selected journals from publishers that Jeffrey Beall identified as predatory in his widely accepted list of predatory publishers. We also referenced journals removed from the Directory of Open Access Journals in 2013. In addition, we systematically identified specific behaviors that contradicted industry standards and best practices. This process led us to 65 behavioral indicators of deception that were frequent or common in journals identified as deceptive. We then weighted each indicator and put it into 1 of 2 categories: indicators directly reflecting deception were weighted heavily and behaviors tending to coincide with deception were weighted lightly. The weights were carefully structured to prevent bias against new or inexperienced journals. We then created a rubric and applied it to data collected about each journal. This produced a weighted score whose magnitude increased with the probability that a journal was engaging in deceptive behavior.
Results
We applied the rubric to 2 different sets of journals: 1192 randomly selected journals from 57 publishers on Jeffrey Beall’s list and 100 journals from Cabell’s database. Of the 100 journals from Cabell’s, we randomly selected 50 journals from the top tier and bottom tier. Of the 1192 journals from Jeffrey Beall’s list, 1114 (93.5%) were flagged as deceptive by our methodology and 78 (6.5%) journals were not flagged as deceptive. Only 6 (10.5%) of the 57 total publishers examined had some journals flagged as deceptive and some not flagged. Our methodology did not identify any of the journals in the Cabell’s set as deceptive.
Conclusions
The results led us to conclude that our methodology was effective in objectively identifying deceptive journals. The 78 journals on Jeffrey Beall’s list not identified by our methodology indicated the potential need for additional factors or tighter tolerances. However, any tightening of tolerances must not identify legitimate journals as deceptive. Further study into the methodology could be conducted to see if it identifies false positives from a set of journals that Jeffrey Beall evaluated but did not indicate as predatory.
1Cabell’s International, Beaumont, TX, USA, kathleen.berryman@cabells.com
Conflict of Interest Disclosures:
Cabell’s International intends to patent the methodology assessed in this paper and use it to create a list of deceptive journals that will be available by subscription.
Funding/Support:
There was no external funding for this study. Contributions of staff time and resources came from Cabell’s International.