Abstract
Automating the Detection of Promotional (Hype) Language in Biomedical Research
Bojan Batalo,1 Erica K. Shimomoto,1 Neil Millar2
Objective
Promotional language (hype) in biomedical research writing has increased significantly over the past 40 years and has potential to influence readers’ perceptions and evaluations of evidence.1-3 Automatic systems capable of detecting and providing feedback on hype may offer a means to foster more objective reporting. However, the absence of formal guidelines for identifying hype represents a barrier for human annotators and the development of such systems. This pilot study develops formal guidelines for classifying hype and evaluates the application of annotated data to automate hype detection via machine learning.
Design
Annotation guidelines were developed to classify 11 adjectives commonly associated with novelty and potential hype (eg, creative, first, groundbreaking, innovative). Guidelines followed the following hierarchical decisions: (1) does the adjective imply a positive value judgment?; (2) is it hyperbolic?; (3) is the adjective gratuitous (ie, adds little to the content), amplified (strengthened by modifiers), or coordinated (paired with other promotional adjectives)?; and (4) is the broader sentential context promotional? A total of 550 sentences containing the adjectives (50 per adjective) were randomly sampled from National Institutes of Health grant abstracts funded between 2016 and 2020 and independently annotated by a linguist and 2 computer scientists, with disagreements resolved through discussion, providing the criterion standard. Our choice of grant abstracts as the starting point for our research is due to the highly competitive nature of research funding.1 The annotated data were split 80:20 into development and hold-out test sets; machine learning algorithms were trained on the development and evaluated on the test set. Additionally, a human baseline was established by an additional researcher blinded to the annotation guidelines but supplied with a broad definition of hype, manually classifying the test set. Experiments tested 4 traditional text classification methods (multinomial Naive Bayes, multivariate Bernoulli Naive Bayes, latent semantic analysis, and support vector machines), with bag-of-words and averaged global vectors for word representation (GloVe) word embeddings as input features (Table 25-0929). Performance was evaluated using accuracy, weighted precision, recall, and F1 scores.
Results
Excluding 13 sentences due to nonadherence with the guidelines, the final sample comprised 537 reliably annotated sentences (Cohen κ > 0.94), of which 392 were classified as hype. Among machine learning models, GloVe-based support vector machines outperformed others with an accuracy of 79.6%, approximating the human baseline of 82.4%.
Conclusions
The annotation process underscored the subjective nature of assessing promotional language, particularly for context-dependent adjectives like emerging and latest, and the need for refined constructs to capture gradations of promotional language. Despite these limitations, the pilot study indicates the potential for machine learning models trained on well-annotated datasets to contribute to the automated detection of hype. Future steps include modifying the annotation systems and experimenting with large language models under zero/few-shot regimes.
References
1. Millar N, Batalo B, Budgell B. Trends in the use of promotional language (hype) in abstracts of successful National Institutes of Health Grant Applications, 1985-2020. JAMA Netw Open. 2022;5(8):e2228676. doi:10.1001/jamanetworkopen.2022.28676
2. Qiu HS, Peng H, Fosse HB, Woodruff TK, Uzzi B. Use of promotional language in grant applications and grant success. JAMA Netw Open. 2024;7(12):e2448696. doi:10.1001/jamanetworkopen.2024.48696
3. Van Den Besselaar P, Mom C. The effect of writing style on success in grant applications. J Informetr. 2022;16(1):101257. doi:10.1016/j.joi.2022.101257
1National Institute of Advanced Industrial Science and Technology, Japan, bojan.batalo@aist.go.jp; 2University of Tsukuba, Japan.
Conflict of Interest Disclosures
None reported.
Funding/Support
This study was supported by grants 21K02919 and 25K00851 from the Japan Society for the Promotion of Science.
Role of the Funder/Sponsor
The authors confirm that no sponsors or funders influenced the study design, execution, or interpretation of results.
Additional Information
Neil Millar is a secondary corresponding author (millar.neil@u.tsukuba.ac.jp).