Robert Frank,1 Matthew McInnes,2,3 Deborah Levine,4,5 Herbert Kressel,4,5 Julie Jesurum,6 William Petrcich,3 Trevor McGrath,1 Patrick M. Bossuyt7
Are most research findings false but less so in high-impact journals? Variables associated with false results in imaging research are poorly understood. While absolute truth is elusive, high-quality meta-analyses using hierarchical methods produce high-level evidence with minimal random variability in their results, thereby representing a convenient proxy for truth. We evaluated whether journal-level and study-level variables are associated with the distance between primary study results and summary estimates from meta-analyses.
In this meta-research study, PubMed was searched for diagnostic accuracy meta-analyses using hierarchical methods, published in imaging journals between January 2005 and April 2016. Data were extracted for each meta-analysis and its included primary studies, including study demographic information, journal Impact Factor, journal cited half-life, Standards for Reporting Diagnostic accuracy studies (STARD) endorsement, citation rate, publication date, sample size, sensitivity, and specificity. Meta-analyses were excluded for failing to report both primary and summary accuracy estimates. Primary studies were divided into 2 groups for each variable assessed; groups were defined based on first publication vs subsequent publications on a topic, publication before vs after STARD introduction, presence vs absence of STARD endorsement, or by median split. The mean absolute deviation of primary study estimates from the corresponding summary estimates for sensitivity and specificity was compared between groups for each variable. Analyses were performed using a model combining a γ distribution for absolute deviations greater than 0 with an estimated probability that the absolute deviation is 0. Means and 95% CIs were obtained using bootstrap resampling. P values were calculated using a t test. The threshold for significance was defined as P < .004 after Bonferroni correction (.05/12) to mitigate bias owing to multiple comparisons.
Ninety-eight meta-analyses containing 1458 primary studies met inclusion criteria. There was substantial variability in deviations from the summary estimate between paired groups, but no variable demonstrated a significant association with proximity of primary study diagnostic accuracy estimates to the pooled estimates from their corresponding meta-analyses (P > .004 in all comparisons)(Table).
Many variables considered important when selecting imaging diagnostic accuracy literature to guide clinical decisions are not associated with results that are more reflective of the truth as established by meta-analyses. The distance between primary study results and summary estimates of diagnostic accuracy is probably not smaller for studies published in higher versus lower Impact Factor journals.
1Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada, firstname.lastname@example.org; 2Department of Radiology, The Ottawa Hospital, University of Ottawa, Ottawa, ON, Canada; 3Ottawa Hospital Research Institute, Ottawa, ON, Canada; 4Department of Radiology, Beth Israel Deaconess Medical Center, Boston, MA, USA; 5Radiology Editorial Office, Boston, MA, USA; 6Harvard University, Boston, MA, USA; 7Clinical Epidemiology and Biostatistics, Bioinformatics Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands
Conflict of Interest Disclosures:
This work was supported in part by stipends from the Ottawa Hospital Department of Radiology and the Undergraduate Research Opportunity Program.
Role of the Funder/Sponsor:
The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract.