A Framework for Evaluation and Use of Automated Scoring

• Published in: Educational measurement, issues and practice Vol. 31; no. 1; pp. 2 - 13
• Main Authors: Williamson, David M., Xi, Xiaoming, Breyer, F. Jay
• Format: Journal Article
• Language: English
• Published: Malden, USA Blackwell Publishing Inc 01.03.2012; Wiley-Blackwell; Wiley Subscription Services, Inc
• Subjects: automated scoring; Automation; essay scoring; Evaluation; Guidelines; High Stakes Tests; Interrater Reliability; Measurement; Measurement Techniques; performance testing; Scoring; Validity
• ISSN: 0731-1745; 1745-3992
• DOI: 10.1111/j.1745-3992.2011.00223.x

A framework for evaluation and use of automated scoring of constructed‐response tasks is provided that entails both evaluation of automated scoring as well as guidelines for implementation and maintenance in the context of constantly evolving technologies. Consideration of validity issues and challenges associated with automated scoring are discussed within the framework. The fit between the scoring capability and the assessment purpose, the agreement between human and automated scores, the consideration of associations with independent measures, the generalizability of automated scores as implemented in operational practice across different tasks and test forms, and the impact and consequences for the population and subgroups are proffered as integral evidence supporting use of automated scoring. Specific evaluation guidelines are provided for using automated scoring to complement human scoring for tests used for high‐stakes purposes. These guidelines are intended to be generalizable to new automated scoring systems and as existing systems change over time.