Examining the Impacts of Rater Effects in Performance Assessments

• Published in: Applied psychological measurement Vol. 43; no. 2; pp. 159 - 171
• Main Author: Wind, Stefanie A.
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01.03.2019
• Subjects: Rasch measurement theory; performance assessment; rater effects
• ISSN: 0146-6216; 1552-3497
• DOI: 10.1177/0146621618789391

Rater effects such as severity, centrality, and misfit are recurrent concerns in performance assessments. Despite their persistence in operational assessment settings and frequent discussion in research, researchers have not fully explored the impacts of rater effects as they relate to estimates of student achievement. The purpose of this study is to explore the impacts of rater severity, centrality, and misfit on student achievement estimates and on classification decisions. The results suggest that these three types of rater effects have substantial impacts on estimates of student achievement and on classification decisions that impact the fairness of rater-mediated assessments. Accordingly, it is essential that researchers and practitioners evaluate ratings across all stages of rater-mediated assessment procedures, including rater training and operational scoring.