Seven reliability indices for high-stakes decision making: Description, selection, and simple calculation

• Published in: Psychology in the schools Vol. 48; no. 10; pp. 1064 - 1075
• Main Authors: Smith, Stacey L., Vannest, Kimberly J., Davis, John L.
• Format: Journal Article
• Language: English
• Published: Hoboken Wiley Subscription Services, Inc., A Wiley Company 01.12.2011; John Wiley & Sons, Inc; Wiley
• Subjects: Biological and medical sciences; Classification; Clinical assessment; Computation; Data Analysis; Decision Making; Educational psychology; Elementary Education; Flow Charts; Fundamental and applied biological sciences. Psychology; Grade 5; High Stakes Tests; Internet; Interrater Reliability; Monte Carlo Methods; Orientation. Evaluation; Psychology. Psychoanalysis. Psychiatry; Psychology. Psychophysiology; Reliability; School Psychology; Scores; Human; School psychologist; Evaluation; Methodology; Decision making; Professional practice; Psychometrics; Measurement method; Test reliability; School environment
• ISSN: 0033-3085; 1520-6807
• DOI: 10.1002/pits.20610

The reliability of data is a critical issue in decision‐making for practitioners in the school. Percent Agreement and Cohen's kappa are the two most widely reported indices of inter‐rater reliability, however, a recent Monte Carlo study on the reliability of multi‐category scales found other indices to be more trustworthy given the type of data and number of categories. This manuscript presents defensible decision steps, methods, and rationale for selecting and calculating inter‐rater reliability for practitioners. In addition to using screen shots from readily available online programs for calculation such as Excel and Vassar College (Lowry, 2010), decision guides and a flow chart are presented in several figures and tables for easy reference. © 2011 Wiley Periodicals, Inc.