Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter‐rater reliability

• Published in: British journal of mathematical & statistical psychology Vol. 77; no. 3; pp. 651 - 671
• Main Authors: Bartoš, František, Martinková, Patrícia
• Format: Journal Article
• Language: English
• Published: England 01.11.2024
• Subjects: Computer Simulation; error rate; False Positive Reactions; Humans; mixed‐effect models; Models, Statistical; Observer Variation; Peer Review - methods; Probability; rating; Reproducibility of Results; Type I error; Type II error; rating; Type II error; Type I error; mixed‐effect models; error rate

Inter‐rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the applicant is either selected or not. This final outcome is...