On the Information Obtainable from Comparative Judgments

• Published in: Psychometrika Vol. 87; no. 4; pp. 1439 - 1472
• Main Author: Bürkner, Paul-Christian
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.12.2022; Springer Nature B.V
• Subjects: Assessment; Bayes Theorem; Bayesian analysis; Behavioral Science and Psychology; Humanities; Humans; Item response theory; Judgment; Law; Likert Scales; Mathematical models; Personality; Personality Tests; Psychological tests; Psychology; Psychometrics; Rating Scales; Statistical Theory and Methods; Statistics for Social Sciences; Test Construction; Testing and Evaluation; Theory and Methods; ordinal models; faking resistance; forced-choice format; Thurstonian IRT models; optimal experimental design; Comparative judgments
• ISSN: 0033-3123; 1860-0980; 1860-0980
• DOI: 10.1007/s11336-022-09843-z

Personality tests employing comparative judgments have been proposed as an alternative to Likert-type rating scales. One of the main advantages of a comparative format is that it can reduce faking of responses in high-stakes situations. However, previous research has shown that it is highly difficult to obtain trait score estimates that are both faking resistant and sufficiently accurate for individual-level diagnostic decisions. With the goal of contributing to a solution, I study the information obtainable from comparative judgments analyzed by means of Thurstonian IRT models. First, I extend the mathematical theory of ordinal comparative judgments and corresponding models. Second, I provide optimal test designs for Thurstonian IRT models that maximize the accuracy of people’s trait score estimates from both frequentist and Bayesian statistical perspectives. Third, I derive analytic upper bounds for the accuracy of these trait estimates achievable through ordinal Thurstonian IRT models. Fourth, I perform numerical experiments that complement results obtained in earlier simulation studies. The combined analytical and numerical results suggest that it is indeed possible to design personality tests using comparative judgments that yield trait scores estimates sufficiently accurate for individual-level diagnostic decisions, while reducing faking in high-stakes situations. Recommendations for the practical application of comparative judgments for the measurement of personality, specifically in high-stakes situations, are given.