Item Complexity: A Neglected Psychometric Feature of Test Items?

• Published in: Psychometrika Vol. 87; no. 4; pp. 1195 - 1213
• Main Authors: Bolt, Daniel M., Liao, Xiangyi
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.12.2022; Springer Nature B.V
• Subjects: Assessment; Asymmetry; Behavioral Science and Psychology; Correlation; Humanities; Item response theory; Language Acquisition; Law; Problem Solving; Psychology; Psychometrics; Quantitative psychology; Sensitivity analysis; Statistical Theory and Methods; Statistics for Social Sciences; Surveys and Questionnaires; Test Items; Testing and Evaluation; Theory and Methods; Theory and Methods (T&M); Item complexity; Item response theory
• ISSN: 0033-3123; 1860-0980; 1860-0980
• DOI: 10.1007/s11336-022-09842-0

Despite its frequent consideration in test development, item complexity receives little attention in the psychometric modeling of item response data. In this address, I consider how variability in item complexity can be expected to emerge in the form of item characteristic curve (ICC) asymmetry, and how such effects may significantly influence applications of item response theory, especially those that assume interval-level properties of the latent proficiency metric and groups that vary substantially in mean proficiency. One application is the score gain deceleration phenomenon often observed in vertical scaling contexts, especially in subject areas like math or second language acquisition. It is demonstrated how the application of symmetric IRT models in the presence of complexity-induced positive ICC asymmetry can be a likely cause. A second application concerns the positive correlation between DIF and difficulty commonly seen in verbal proficiency (and other subject area) tests where problem-solving complexity is minimal and proficiency-related guessing effects are likely pronounced. Here we suggest negative ICC asymmetry as a probable cause and apply sensitivity analyses to demonstrate the ease with which such correlations disappear when allowing for negative ICC asymmetry. Unfortunately, the presence of systematic forms of ICC asymmetry is easily missed due to the considerable flexibility afforded by latent trait metrics in IRT. Speculation is provided regarding other applications for which attending to ICC asymmetry may prove useful.