Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters

• Published in: Computer methods and programs in biomedicine Vol. 215; p. 106654
• Main Authors: Golzarri-Arroyo, Lilian, Dickinson, Stephanie L., Jamshidi-Naeini, Yasaman, Zoh, Roger S., Brown, Andrew W., Owora, Arthur H., Li, Peng, Oakes, J. Michael, Allison, David B.
• Format: Journal Article
• Language: English
• Published: Ireland Elsevier B.V 01.03.2022
• Subjects: Cluster Analysis; Computer Simulation; Humans; Linear Models; Research Design; Sample Size
• ISSN: 0169-2607; 1872-7565; 1872-7565
• DOI: 10.1016/j.cmpb.2022.106654

•The type I error rate is inflated under scenarios of a small number of clusters per treatment in a cluster randomized trials when using parametric bootstrap.•When analyzing cluster randomized trials, the pbkrtest package is limited in performance as it resamples the observations ignoring clusters.•We want to highlight that while the well-known nesting/degrees of freedom issue undermines p-values when k <= 20, we show here that small number of clusters and small ICC inflate type I error rates, setting the nesting/df issue aside. Cluster randomized controlled trials (cRCTs) are increasingly used but must be analyzed carefully. We conducted a simulation study to evaluate the validity of a parametric bootstrap (PB) approach with respect to the empirical type I error rate for a cRCT with binary outcomes and a small number of clusters. We simulated a case study with a binary (0/1) outcome, four clusters, and 100 subjects per cluster. To compare the validity of the test with respect to error rate, we simulated the same experiment with K=10, 20, and 30 clusters, each with 2,000 simulated datasets. To test the null hypothesis, we used a generalized linear mixed model including a random intercept for clusters and obtained p-values based on likelihood ratio tests (LRTs) using the parametric bootstrap method as implemented in the R package “pbkrtest”. The PB test produced error rates of 9.1%, 5.5%, 4.9%, and 5.0% on average across all ICC values for K=4, K=10, K=20, and K=30, respectively. The error rates were higher, ranging from 9.1% to 36.5% for K=4, in the models with singular fits (i.e., ignoring clustering) because the ICC was estimated to be zero. Using the parametric bootstrap for cRCTs with a small number of clusters results in inflated error rates and is not valid