Outliers in Questionnaire Data: Can They Be Detected and Should They Be Removed?

• Published in: Journal of Educational and Behavioral Statistics Vol. 36; no. 2; pp. 186 - 212
• Main Authors: Zijlstra, Wobbe P., van der Ark, L. Andries, Sijtsma, Klaas
• Format: Journal Article Book Review
• Language: English
• Published: Los Angeles, CA SAGE Publications 01.04.2011; American Educational Research Association
• Subjects: Bias; Coefficients; Data; Datasets; Methods; Observational research; Outliers; Proportions; Questionnaires; Rating Scales; Sampling bias; Standard deviation; Statistical Bias; Statistical variance; Statistics; Validity; outlier detection in questionnaire data; outlier detection methods for questionnaire data; contaminant observations; discordancy testing; suspected patterns of item scores
• ISSN: 1076-9986; 1935-1054
• DOI: 10.3102/1076998610366263

Outliers in questionnaire data are unusual observations, which may bias statistical results, and outlier statistics may be used to detect such outliers. The authors investigated the effect outliers have on the specificity and the sensitivity of each of six different outlier statistics. The Mahalanobis distance and the item-pair based outlier statistics were found to have the best combination of specificity and sensitivity. Next, it was investigated how outliers influenced the bias in the percentile rank score, Cronbach's alpha, and the validity coefficient. Outliers due to random responding and faking produced considerable bias, and outliers due to extreme responding produced little bias. Finally, the influence of removing discordant observations on bias was studied. Removing observations due to random responding identified by means of the Mahalanobis distance, the local outlier factor, and the item-pair based outlier statistic reduced bias.