Abstract
Misidentification of Scanning Electron Microscope Instruments in the Peer-Reviewed Materials Science and Engineering Literature
Reese A. K. Richardson,1,2 Jeonghyun Moon,1 Spencer S. Hong,1,3 Luis A. Nunes Amaral1,2,4,5,6
Objective
Materials science and engineering (MSE) research has, for the most part, escaped the doubts raised about the reliability of scientific literature by recent large-scale replication studies in psychology and cancer biology. However, users on postpublication peer review sites have recently identified dozens of articles where the make and model of the scanning electron microscope (SEM) listed in the text does not match the instrument’s metadata visible in the images in the published article.
Design
To systematically investigate this potential risk to the MSE literature, we developed a semiautomated approach using optical character recognition to scan published figures for instrumental metadata banners and check the information contained in these banners on instrument manufacturer and model against the SEM instrument identified in the text. We validated our approach by evaluating model performance on known instances of SEM misidentification and manual reevaluation of novel instances of SEM misidentification returned by the pipeline. This analysis was performed in January 2024.
Results
Starting from a set of 1,067,102 articles published since 2010 (as indexed by OpenAlex, downloaded on February 27, 2023) in 50 MSE journals (selected according to the publisher’s permissiveness of text and data mining and to represent a broad range of subdisciplines and impact factors), we identified 11,314 articles for which SEM make and model could be identified in an image’s metadata. For 2400 of those articles (21.2%), the image metadata did not match the SEM manufacturer or model listed in the text, and for another 2799 (24.7%), at least some of the instruments used in the study were not reported. We found that articles with SEM misidentification were more likely to have existing PubPeer comments than articles without (43 of 2400 [1.8%] vs 984 of 171,646 [0.5%]; P = 9.98 × 10-15) and that electrochemistry articles that featured SEM misidentification, compared with those that did not have SEM misidentification, were more likely to make a crucial error in their determination of optical band gap (41 of 154 [26.6%] vs 100 of 751 [13.3%]; P = 3.35 × 10-5).
Conclusions
Our results suggest that SEM misidentification can be a valuable indicator of irreproducibility in the results of extant scientific articles. We recommend that peer reviewers, before and after publication, remain attentive to the accurate reporting of instruments used in scientific articles. Our selection criteria for journals and the high false-negative rate of our pipeline limit the extent to which our findings are representative of the MSE literature at large. Unexplained textual similarities (eg, the same unlikely typo appearing in dozens of articles) common to many of these articles suggest the involvement of paper mills, organizations that mass produce, sell authorship on, and publish fraudulent scientific manuscripts at scale.
1Department of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, IL US, richardsonr43@gmail.com; 2Department of Molecular Biosciences, Northwestern University, Evanston, IL US; 3Department of Chemical and Biological Engineering, Northwestern University, Evanston, IL US; 4Department of Physics and Astronomy, Northwestern University, Evanston, IL, US; 5Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, US; 6NSF-Simons National Institute for Theory and Mathematics in Biology, Chicago, IL, US.
Conflict of Interest Disclosures
None reported.
Funding/Support
Reese A. K. Richardson was supported in part by the National Institutes of Health Training grant T32GM008449 through Northwestern University’s Biotechnology Training Program; Reese A. K. Richardson gratefully acknowledges funding from the Dr. John N. Nicholson fellowship from Northwestern University and Moderna Inc. Spencer S. Hong gratefully acknowledges support from the Ryan Fellowship and the International Institute for Nanotechnology at Northwestern University; Luis A. Nunes Amaral gratefully acknowledges funding from SCISIPBIO: a data-science approach to evaluating the likelihood of fraud and error in published studies (grant No. 1956338).
Role of the Funder/Sponsor
The funders had no role in the design or execution of this study.