International Congress on
Peer Review and Scientific Publication

Jae Il Shin,^1,2,3 Min Seo Kim,^4,5,6,7 Dong Keon Yon,⁸ Seung Won Lee,⁹ Masoud Rahmati,¹⁰ Marco Solmi,^{11,12,13,14,15} Andre F. Carvalho,¹⁶ Ai Koyanagi,^17,18,19 Lee Smith,²⁰ John P. A. Ioannidis^{21,22,23,24,25}

Objective

Immortal time bias (ITB) is the error in estimating the association between the exposure and the outcome that results from misclassification or exclusion of time intervals. The aim of our study was to estimate the prevalence of ITB in systematic reviews and meta-analyses and to assess the degree to which it contributes to effect size estimates and evidence reversal.

Design

We performed a systematic review of systematic reviews with meta-analyses (SRMA) that only underwent detailed analysis based on ITB. We only included SRMA of observational studies including cohort and case-control studies and excluded those without meta-analysis or SRMA of randomized control trials. We searched PubMed/MEDLINE, Embase, and Cochrane Database of Systematic Reviews from database inception to July 31, 2024. Two authors independently extracted data and evaluated the methodological quality of the systematic reviews. Information on ITB judgment and effect sizes with 95% CIs for individual studies in forest plots were extracted to run reanalysis using generic inverse variance fixed- and random-effects methods. After extracting data, we conducted subgroup analyses by the presence of ITB for all available topics and assessed the influence of ITB on the heterogeneity (I²), changes of evidence, statistical significance of the finding, and altering effect size in favor of intervention/exposure.

Results

Among the 12 systematic reviews with relevant data, there were 25 eligible topics (only 21 could be divided by ITB, as 4 included only studies without ITB). The median (IQR) number of studies included for a topic was 6 (4-10). Among 182 studies among 25 topics, 44.0% (80 studies) were affected by ITB. Among the 21 topics where both studies with ITB and studies without ITB were available, 57.1% (12/21) demonstrated discordant results between ITB subgroups (Figure 25-0976). Evidence reversal occurred in 23.8% (5/21), where overall summary results changed from statistically significant to non–statistically significant or vice versa after excluding studies with ITB. The ratio of effect size (effect sizes pooled from studies with ITB relative to those pooled from studies without ITB) was 0.71 (95% CI, 0.66-0.78), suggesting that the effect sizes from studies with ITB were exaggerated by an average of 29% in favor of the intervention/exposure. Excluding studies involving ITB reduced the heterogeneity (I²) of overall pooled results by 21.4% on average.

Conclusions

We quantitatively captured how far ITB has influenced our knowledge and clinical practices. Given the projected high prevalence and nontrivial influence of ITB, ITB should be considered in studies with survival analyses, and improving reporting standards by researchers as well as collective surveillance from readers, reviewers, and editors is warranted. Future studies should address how ITS may also interact with other trial characteristics and biases in affecting treatment effect estimates.

Affiliations

¹Department of Pediatrics, Yonsei University College of Medicine, Seoul, Republic of Korea, shinji@yuhs.ac; ²Severance Underwood Meta-Research Center, Institute of Convergence Science, Yonsei University, Seoul, Republic of Korea; ³Affiliate in Meta-Research Innovation Center at Stanford, Stanford University, Stanford, CA, US; ⁴Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Samsung Medical Center, Seoul, Republic of Korea; ⁵Medical and Population Genetics and Cardiovascular Disease Initiative, Broad Institute, Cambridge, MA, US; ⁶Cardiovascular Research Center, Massachusetts General Hospital, Boston, MA, US; ⁷Department of Medicine, Harvard Medical School, Boston, MA, US; ⁸Center for Digital Health, Medical Science Research Institute, Kyung Hee University College of Medicine, Seoul, Republic of Korea; ⁹Department of Data Science, Sejong University College of Software Convergence, Seoul, Republic of Korea; ¹⁰Department of Physical Education and Sport Sciences, Faculty of Literature and Human Sciences, Lorestan University, Khoramabad, Iran; ¹¹Department of Psychiatry, University of Ottawa, Ottawa, Ontario, Canada; ¹²Department of Mental Health, The Ottawa Hospital, Ottawa, Ontario, Canada; ¹³Ottawa Hospital Research Institute (OHRI) Clinical Epidemiology Program University of Ottawa, Ottawa, Ontario, Canada; ¹⁴School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada; ¹⁵Department of Child and Adolescent Psychiatry, Charité Universitätsmedizin, Berlin, Germany; ¹⁶Innovation in Mental and Physical Health and Clinical Treatment, Strategic Research Centre, School of Medicine, Barwon Health, Deakin University, Geelong, VIC, Australia; ¹⁷Research and Development Unit, Parc Sanitari Sant Joan de Déu, Universitat de Barcelona, Fundació Sant Joan de Déu, CIBERSAM, Barcelona, Spain; ¹⁸ICREA, Pg. Lluis Companys 23, Barcelona, Spain; ¹⁹Instituto de Salud Carlos III, Centro de Investigación Biomédica en Red de Salud Mental, CIBERSAM, Madrid, Spain; ²⁰The Cambridge Centre for Sport and Exercise Sciences, Anglia Ruskin University, Cambridge, United Kingdom; ²¹Department of Medicine, Stanford University, Stanford, CA, US; ²²Department of Epidemiology and Population Health, Stanford University, Stanford, CA, US; ²³Department of Biomedical Data Science, Stanford University, Stanford, CA, US; ²⁴Department of Statistics, Stanford University, Stanford, CA, US; ²⁵Meta-Research Innovation Center at Stanford, Stanford University, Stanford, CA, US.

Conflict of Interest Disclosures

John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.

Funding/Support

The work of John P. A. Ioannidis is supported by an unrestricted gift from Sue and Bob O’Donnell to Stanford University. Jae Il Shin is supported by the Yonsei Faculty Fellowship, funded by Lee Youn Jae.

Role of the Funder/Sponsor

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Additional Information

John P. A. Ioannidis is a cocorresponding author (jioannid@stanford.edu).