The Impact of Model Parameterization and Estimation Methods on Tests of Measurement Invariance With Ordered Polytomous Data

• Published in: Educational and psychological measurement Vol. 78; no. 2; pp. 272 - 296
• Main Authors: Koziol, Natalie A., Bovaird, James A.
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01.04.2018; SAGE PUBLICATIONS, INC
• Subjects: Chi-square test; Comparative Analysis; Computation; Construct Validity; Educational Research; Error of Measurement; Factor Analysis; Goodness of Fit; Hypotheses; Least Squares Statistics; Maximum Likelihood Statistics; Measurement Techniques; Monte Carlo Methods; Monte Carlo simulation; Researchers; Statistical Analysis; Studies; Test Bias; Tests; Validity; measurement invariance; robust maximum likelihood; limited information estimation; Tobit model; categorical confirmatory factor analysis; polytomous data; differential item functioning
• ISSN: 0013-1644; 1552-3888; 1552-3888
• DOI: 10.1177/0013164416683754

Evaluations of measurement invariance provide essential construct validity evidence—a prerequisite for seeking meaning in psychological and educational research and ensuring fair testing procedures in high-stakes settings. However, the quality of such evidence is partly dependent on the validity of the resulting statistical conclusions. Type I or Type II errors can render measurement invariance conclusions meaningless. The present study used Monte Carlo simulation methods to compare the effects of multiple model parameterizations (linear factor model, Tobit factor model, and categorical factor model) and estimators (maximum likelihood [ML], robust maximum likelihood [MLR], and weighted least squares mean and variance-adjusted [WLSMV]) on the performance of the chi-square test for the exact-fit hypothesis and chi-square and likelihood ratio difference tests for the equal-fit hypothesis for evaluating measurement invariance with ordered polytomous data. The test statistics were examined under multiple generation conditions that varied according to the degree of metric noninvariance, the size of the sample, the magnitude of the factor loadings, and the distribution of the observed item responses. The categorical factor model with WLSMV estimation performed best for evaluating overall model fit, and the categorical factor model with ML and MLR estimation performed best for evaluating change in fit. Results from this study should be used to inform the modeling decisions of applied researchers. However, no single analysis combination can be recommended for all situations. Therefore, it is essential that researchers consider the context and purpose of their study.