Inter-rater reliability of query/probe-based techniques for measuring situation awareness

• Published in: Ergonomics Vol. 57; no. 7; pp. 959 - 972
• Main Authors: Lau, Nathan, Jamieson, Greg A., Skraaning, Gyrd
• Format: Journal Article
• Language: English
• Published: London Taylor & Francis 03.07.2014; Washington, DC Taylor & Francis LLC
• Subjects: Assessments; Awareness; Biological and medical sciences; Computer Simulation; Empirical Research; Ergonomics; Ergonomics. Human factors; experiments; full-scope simulator; Fundamental and applied biological sciences. Psychology; Human; Humans; Judgment; Keys; Knowledge; Measurement; nuclear; Nuclear power plants; Nuclear Power Plants - standards; Observer Variation; Occupational psychology; Operators; performance assessment; process control; Psychology. Psychoanalysis. Psychiatry; Psychology. Psychophysiology; Psychometrics - methods; Query processing; ratings; Reliability; Reproducibility of Results; Simulation; subject-matter-experts; Task Performance and Analysis; Performance evaluation; Human; Query; Process control; Awareness; Expert; experiments; Psychometrics; subject-matter-experts; Test reliability; Interrater agreement; Ergonomics; Nuclear power plant; ratings; performance assessment; full-scope simulator; nuclear; Simulator; process control

Query- or probe-based situation awareness (SA) measures sometimes rely on process experts to evaluate operator actions and system states when used in representative settings. This introduces variability of human judgement into the measurements that require inter-rater reliability assessment. However...