Reliability Theory for Measurements with Variable Test Length, Illustrated with ERN and Pe Collected in the Flanker Task

• Published in: Psychometrika Vol. 89; no. 4; pp. 1280 - 1303
• Main Authors: Ellis, Jules L., Sijtsma, Klaas, de Groot, Kristel, Groenen, Patrick J. F.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.12.2024; Springer Nature B.V
• Subjects: Assessment; Bayes Theorem; Bayesian analysis; Bayesian Statistics; Behavioral Science and Psychology; Classical test theory; Cronbach's alpha; Electroencephalography - methods; Event-related potentials; Evoked Potentials - physiology; Female; Generalizability Theory; Humanities; Humans; Law; Male; Methods; Psychology; Psychometrics; Psychometrics - methods; Psychophysiology; Quantitative psychology; Random variables; Reaction Time - physiology; Reliability; Reproducibility of Results; Statistical Theory and Methods; Statistics for Social Sciences; Test Length; Test Theory; Testing and Evaluation; Theory & Methods; ERN; Flanker Task; classical test theory; event-related potentials; Pe; reliability
• ISSN: 0033-3123; 1860-0980; 1860-0980
• DOI: 10.1007/s11336-024-09982-5

In psychophysiology, an interesting question is how to estimate the reliability of event-related potentials collected by means of the Eriksen Flanker Task or similar tests. A special problem presents itself if the data represent neurological reactions that are associated with some responses (in case of the Flanker Task, responding incorrectly on a trial) but not others (like when providing a correct response), inherently resulting in unequal numbers of observations per subject. The general trend in reliability research here is to use generalizability theory and Bayesian estimation. We show that a new approach based on classical test theory and frequentist estimation can do the job as well and in a simpler way, and even provides additional insight to matters that were unsolved in the generalizability method approach. One of our contributions is the definition of a single, overall reliability coefficient for an entire group of subjects with unequal numbers of observations. Both methods have slightly different objectives. We argue in favor of the classical approach but without rejecting the generalizability approach.