A robust threshold t linear mixed model for subgroup identification using multivariate T distributions

• Published in: Computational statistics Vol. 38; no. 1; pp. 299 - 326
• Main Authors: Zhang, Rui, Qin, Guoyou, Tu, Dongsheng
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.03.2023; Springer Nature B.V
• Subjects: Algorithms; Biomarkers; Colorectal cancer; Economic Theory/Quantitative Economics/Mathematical Methods; Heterogeneity; Identification; Identification methods; Laboratories; Mathematics and Statistics; Monoclonal antibodies; Multivariate analysis; Normal distribution; Original Paper; Outliers (statistics); Performance evaluation; Probability and Statistics in Computer Science; Probability Theory and Stochastic Processes; Public health; Robustness; Simulation; Statistics; Subgroups; ECM algorithm; Subgroup identification; Longitudinal data; Threshold Model; T distribution
• ISSN: 0943-4062; 1613-9658
• DOI: 10.1007/s00180-022-01229-0

Subgroup identification has emerged as a popular statistical tool to access the heterogeneity in treatment effects based on specific characteristics of patients. Recently, a threshold linear mixed-effects model was proposed to identify a subgroup of patients who may benefit from treatment concerning longitudinal outcomes based on whether a continuous biomarker exceeds an unknown cut-point. This model assumes, however, normal distributions to both random effects and error terms and, therefore, may be sensitive to outliers in the longitudinal outcomes. In this paper, we propose a robust subgroup identification method for longitudinal data by developing a robust threshold t linear mixed-effects model, where random effects and within-subject errors follow a multivariate t distribution, with unknown degree-of-freedoms. The likelihood function is, however, difficult to directly maximize because the density function of a non-central t distribution is too complicated to compute and an indicator function is included in the definition of the mode. We, therefore, propose a smoothed expectation conditional maximization algorithm based on a gamma-normal hierarchical structure and the smooth approximation of an indicator function to make inferences on the parameters in the model. Simulation studies are conducted to investigate the performance and robustness of the proposed method. As an application, the proposed method is used to identify a subgroup of patients with advanced colorectal cancer who may have a better quality of life when treated by cetuximab.