Bag of little bootstraps for massive and distributed longitudinal data

• Published in: Statistical analysis and data mining Vol. 15; no. 3; pp. 314 - 321
• Main Authors: Zhou, Xinkai, Zhou, Jin J., Zhou, Hua
• Format: Journal Article
• Language: English
• Published: Hoboken Wiley Subscription Services, Inc., A Wiley Company 01.06.2022; Wiley Subscription Services, Inc
• Subjects: bags of little bootstraps; big data; Data analysis; Datasets; EMR; linear mixed models; longitudinal data; parallel and distributed computing; Personal computers; Statistical analysis; Statistical inference; Statistical methods; Variance; parallel and distributed computing; EMR; linear mixed models; big data; bags of little bootstraps; longitudinal data
• ISSN: 1932-1864; 1932-1872
• DOI: 10.1002/sam.11563

Linear mixed models are widely used for analyzing longitudinal datasets, and the inference for variance component parameters relies on the bootstrap method. However, health systems and technology companies routinely generate massive longitudinal datasets that make the traditional bootstrap method infeasible. To solve this problem, we extend the highly scalable bag of little bootstraps method for independent data to longitudinal data and develop a highly efficient Julia package MixedModelsBLB.jl. Simulation experiments and real data analysis demonstrate the favorable statistical performance and computational advantages of our method compared to the traditional bootstrap method. For the statistical inference of variance components, it achieves 200 times speedup on the scale of 1 million subjects (20 million total observations), and is the only currently available tool that can handle more than 10 million subjects (200 million total observations) using desktop computers.