Adaptive Pairwise Comparison for Educational Measurement

Pairwise comparison is becoming increasingly popular as a holistic measurement method in education. Unfortunately, many comparisons are required for reliable measurement. To reduce the number of required comparisons, we developed an adaptive selection algorithm (ASA) that selects the most informativ...

Full description

Saved in:
Bibliographic Details
Published inJournal of educational and behavioral statistics Vol. 45; no. 3; pp. 316 - 338
Main Authors Crompvoets, Elise A. V., Béguin, Anton A., Sijtsma, Klaas
Format Journal Article
LanguageEnglish
Published Los Angeles, CA SAGE Publications 01.06.2020
American Educational Research Association
Subjects
Accuracy
Algorithms
Comparative Analysis
Educational tests & measurements
Error of Measurement
Mathematics
Measurement
Reliability
Statistical Analysis
pairwise comparison
adaptive measurement
comparative judgment
holistic measurement
Online AccessGet full text

Cover

More Information
Summary:Pairwise comparison is becoming increasingly popular as a holistic measurement method in education. Unfortunately, many comparisons are required for reliable measurement. To reduce the number of required comparisons, we developed an adaptive selection algorithm (ASA) that selects the most informative comparisons while taking the uncertainty of the object parameters into account. The results of the simulation study showed that, given the number of comparisons, the ASA resulted in smaller standard errors of object parameter estimates than a random selection algorithm that served as a benchmark. Rank order accuracy and reliability were similar for the two algorithms. Because the scale separation reliability (SSR) may overestimate the benchmark reliability when the ASA is used, caution is required when interpreting the SSR.
ISSN:1076-9986
1935-1054
DOI:10.3102/1076998619890589