Effects of Discontinue Rules on Psychometric Properties of Test Scores

• Published in: Psychometrika Vol. 84; no. 1; pp. 147 - 163
• Main Authors: von Davier, Matthias, Cho, Youngmi, Pan, Tianshu
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.03.2019; Springer Nature B.V
• Subjects: Adaptive Testing; Assessment; Behavioral Science and Psychology; Children; Computer Simulation; Data Interpretation, Statistical; Educational Testing; Humanities; Humans; Intelligence; Intelligence Tests; Item response theory; Law; Psychology; Psychometrics; Psychometrics - methods; Quantitative psychology; Statistical Theory and Methods; Statistics for Social Sciences; Stochasticity; Test Length; Testing and Evaluation; DIF; discontinue rule; ignorability; local dependency; bias
• ISSN: 0033-3123; 1860-0980
• DOI: 10.1007/s11336-018-09652-3

This paper provides results on a form of adaptive testing that is used frequently in intelligence testing. In these tests, items are presented in order of increasing difficulty. The presentation of items is adaptive in the sense that a session is discontinued once a test taker produces a certain number of incorrect responses in sequence, with subsequent (not observed) responses commonly scored as wrong. The Stanford-Binet Intelligence Scales (SB5; Riverside Publishing Company, 2003 ) and the Kaufman Assessment Battery for Children (KABC-II; Kaufman and Kaufman, 2004 ), the Kaufman Adolescent and Adult Intelligence Test (Kaufman and Kaufman 2014 ) and the Universal Nonverbal Intelligence Test (2nd ed.) (Bracken and McCallum 2015 ) are some of the many examples using this rule. He and Wolfe (Educ Psychol Meas 72(5):808–826, 2012 . https://doi.org/10.1177/0013164412441937 ) compared different ability estimation methods in a simulation study for this discontinue rule adaptation of test length. However, there has been no study, to our knowledge, of the underlying distributional properties based on analytic arguments drawing on probability theory, of what these authors call stochastic censoring of responses. The study results obtained by He and Wolfe (Educ Psychol Meas 72(5):808–826, 2012 . https://doi.org/10.1177/0013164412441937 ) agree with results presented by DeAyala et al. (J Educ Meas 38:213–234, 2001 ) as well as Rose et al. (Modeling non-ignorable missing data with item response theory (IRT; ETS RR-10-11), Educational Testing Service, Princeton, 2010 ) and Rose et al. (Psychometrika 82:795–819, 2017 . https://doi.org/10.1007/s11336-016-9544-7 ) in that ability estimates are biased most when scoring the not observed responses as wrong. This scoring is used operationally, so more research is needed in order to improve practice in this field. The paper extends existing research on adaptivity by discontinue rules in intelligence tests in multiple ways: First, an analytical study of the distributional properties of discontinue rule scored items is presented. Second, a simulation is presented that includes additional scoring rules and uses ability estimators that may be suitable to reduce bias for discontinue rule scored intelligence tests.