Effect Estimates for the Same Outcomes Designated as Primary vs Secondary in Randomized Clinical Trials: A Meta-Research Study
Abstract
Yuanxi Jia,1 Yiwen Jiang,2 Karen A. Robinson,3 Jinling Tang4
Objective
Researchers conducting randomized clinical trials (RCTs) might focus more on primary outcomes than secondary outcomes. Therefore, when the design and conduct of an RCT were manipulated for positive results, the effect size of the primary outcome may be overestimated more than that of the secondary outcomes. Among RCTs with similar design, the discrepant effect estimates of the same outcome between those designating it as a primary outcome (PO-RCTs) and others designating it as a secondary outcome (SO-RCTs) may indicate the impact of bias. This study aimed to compare the effect estimates of the same outcomes between PO-RCTs and SO-RCTs. PO-RCTs were hypothesized to produce higher effect estimates than SO-RCTs.
Design
This meta-research study using a matched cohort design included 153 meta-analyses assessing the efficacy/effectiveness of health interventions published in the Cochrane Database of Systematic Reviews between 2021 and 2023. A total of 1073 RCTs with parallel design, individual randomization, superiority tests, recruitment in or after 2006, and negative control (placebo, sham, no intervention, waiting list, minimal intervention, attention control, or standard care) were included. Within each meta-analysis, RCTs designating the meta-analyzed outcome as primary outcome were classified as PO-RCTs (553 RCTs), while those designating the meta-analyzed outcome as secondary outcome were classified as SO-RCTs (520 RCTs). RCTs that did not define a primary outcome were excluded. The PO-RCTs were compared with the SO-RCTs using 2-stage random-effect meta-analyses: the effect estimates from RCTs were transformed into odds ratios (ORs); within each meta-analysis, the ORs from PO-RCTs and SO-RCTs were compared as a ratio of ORs (ROR); and the RORs across meta-analyses were combined. The primary analyses were conducted among all RCTs and prospectively registered RCTs. The risk of bias for blinding was assessed using the Cochrane Risk of Bias Tool.
Results
Among 1073 RCTs from 153 meta-analyses, PO-RCTs produced ORs 1.27 (95% CI, 1.14-1.42; I2 = 43.6%) times higher than SO-RCTs. Among 372 prospectively registered RCTs from 122 meta-analyses, PO-RCTs produced ORs 1.21 (95% CI, 1.03-1.41; I2 = 22.3%) times higher than SO-RCTs. When restricted to RCTs with high/unclear risk in performance bias, detection bias, or both types of bias, PO-RCTs produced ORs 1.48 (95% CI, 1.26-1.75), 1.44 (95% CI, 1.21-1.71), and 1.60 (95% CI, 1.30-1.96) times higher than SO-RCTs, respectively. When restricted to RCTs with low risk in performance bias, detection bias, or both types of bias, PO-RCTs produced ORs 1.08 (95% CI, 0.90-1.28), 1.04 (95% CI, 0.87-1.24), and 1.06 (95% CI, 0.90-1.26) times higher than SO-RCTs, respectively.
Conclusions
The effect estimate for an outcome could be 27% larger when designated as a primary outcome than a secondary outcome in RCTs, which may be partially attributed to bias. Researchers conducting systematic reviews may need to note whether an outcome was designated as primary or secondary in RCTs, perform sensitivity analyses, and interpret results considering the impact of potential bias.
1Yong Loo Lin School of Medicine, National University of Singapore, Singapore, yx.jia@nus.edu.sg; 2Shenzhen Institute of Advanced Technology, Shenzhen, China; 3School of Medicine, Johns Hopkins University, Baltimore, US; 4Shenzhen University of Advanced Technology, Shenzhen, China.