Good reasons for high variability (low inter-rater reliability) in performance assessment: Toward a fuzzy logic model

• Published in: International journal of industrial ergonomics Vol. 44; no. 5; pp. 685 - 696
• Main Authors: Roth, Wolff-Michael, Mavin, Timothy J., Munro, Ian
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01.09.2014; Elsevier BV
• Subjects: Assessments; Aviation; Crews; Ergonomics; Fuzzy; Fuzzy logic; Fuzzy logic model; High-risk industry; Inter-rater reliability; Mathematical models; Performance assessment; Performance evaluation; Pilots; Ratings; Reliability; Research design; Risk; Think-aloud protocol; Variables; Inter-rater reliability; Fuzzy logic model; Think-aloud protocol; Aviation; High-risk industry; Performance assessment

Regular performance assessment is an integral part of (high-) risk industries. Past research shows, however, that in many fields, inter-rater reliabilities tend to be moderate to low. This study was designed to investigate the variability of performance assessment in a naturalistic setting in aviati...