A sequential Moken scaling approach to evaluate response quality in survey research

• Published in: Behavior research methods Vol. 56; no. 3; pp. 2273 - 2291
• Main Authors: Wind, Stefanie A., Lugu, Benjamin, Wang, Yurou
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.03.2024; Springer Nature B.V
• Subjects: Behavioral Science and Psychology; Cognitive Psychology; Computer Simulation; Humans; Psychology; Psychometrics - methods; Research Design; Surveys; Surveys and Questionnaires; Survey research; Carelessness; Nonparametric item response theory; Mokken scale analysis
• ISSN: 1554-3528; 1554-3528
• DOI: 10.3758/s13428-023-02147-9

Careless responding, where participants do not fully engage with item content, is pervasive in survey research. Left undetected, carelessness can compromise the interpretation and use of survey results, including information about participant locations on the construct, item difficulty, and the psychometric quality of the instrument. We present and illustrate a sequential procedure for evaluating response quality in survey research using indicators from Mokken scale analysis (MSA). We use a real data illustration and a simulation study to compare a sequential procedure to a standalone procedure. We also consider how identifying and removing responses with evidence of poor measurement properties affects item quality indicators. Results suggest that the sequential procedure was effective in identifying potentially problematic response patterns that may not always be captured by traditional methods for identifying careless responders but was not always sensitive to specific carelessness patterns. We discuss implications for research and practice.