The Accuracy of Computerized Adaptive Testing in Heterogeneous Populations: A Mixture Item-Response Theory Analysis

• Published in: PloS one Vol. 11; no. 3; p. e0150563
• Main Authors: Sawatzky, Richard, Ratner, Pamela A, Kopec, Jacek A, Wu, Amery D, Zumbo, Bruno D
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 01.03.2016; Public Library of Science (PLoS)
• Subjects: Accuracy; Activities of daily living; Analysis; Bias; Computer simulation; Data processing; Diagnosis, Computer-Assisted; Equivalence; Female; Heterogeneity; Humans; Item response theory; Male; Mathematical models; Measurement methods; Medicine and Health Sciences; Middle Aged; Models, Statistical; Monte Carlo Method; Monte Carlo simulation; Multivariate analysis; Pain; Pain Measurement - methods; Pain Measurement - standards; People and Places; Physical Sciences; Psychological Theory; Research and Analysis Methods; Research methodology; Researchers; Simulation; Simulation analysis; Structural equation modeling; Surveys and Questionnaires - standards; Treatment Outcome; Canada; New Jersey
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0150563

Computerized adaptive testing (CAT) utilizes latent variable measurement model parameters that are typically assumed to be equivalently applicable to all people. Biased latent variable scores may be obtained in samples that are heterogeneous with respect to a specified measurement model. We examined the implications of sample heterogeneity with respect to CAT-predicted patient-reported outcomes (PRO) scores for the measurement of pain. A latent variable mixture modeling (LVMM) analysis was conducted using data collected from a heterogeneous sample of people in British Columbia, Canada, who were administered the 36 pain domain items of the CAT-5D-QOL. The fitted LVMM was then used to produce data for a simulation analysis. We evaluated bias by comparing the referent PRO scores of the LVMM with PRO scores predicted by a "conventional" CAT (ignoring heterogeneity) and a LVMM-based "mixture" CAT (accommodating heterogeneity). The LVMM analysis indicated support for three latent classes with class proportions of 0.25, 0.30 and 0.45, which suggests that the sample was heterogeneous. The simulation analyses revealed differences between the referent PRO scores and the PRO scores produced by the "conventional" CAT. The "mixture" CAT produced PRO scores that were nearly equivalent to the referent scores. Bias in PRO scores based on latent variable models may result when population heterogeneity is ignored. Improved accuracy could be obtained by using CATs that are parameterized using LVMM.