Comparing correlated kappas by resampling: Is one level of agreement significantly different from another?

• Published in: Journal of psychiatric research Vol. 30; no. 6; pp. 483 - 492
• Main Authors: McKenzie, Dean P., Mackinnon, Andrew J., Péladeau, Normand, Onghena, Patrick, Bruce, Peter C., Clarke, David M., Harrigan, Susan, McGorry, Patrick D.
• Format: Journal Article
• Language: English
• Published: Oxford Elsevier Ltd 01.11.1996; Elsevier
• Subjects: Adult; Biological and medical sciences; Depressive Disorder - diagnosis; Depressive Disorder - psychology; Female; Humans; Male; Mathematical Computing; Medical sciences; Methodology. Experimentation; Monte Carlo Method; Observer Variation; Personality Inventory - statistics & numerical data; Psychology. Psychoanalysis. Psychiatry; Psychometrics - methods; Psychopathology. Psychiatry; Reproducibility of Results; Techniques and methods; Human; Statistical method; Mood disorder; Interrater agreement; Methodology; Diagnosis; Sampling; Comparative study
• ISSN: 0022-3956; 1879-1379
• DOI: 10.1016/S0022-3956(96)00033-7

Researchers comparing diagnostic systems or screening tests frequently need to compare indices of agreement such as the kappa coefficient. While asymptotic methods exist for comparing kappas derived from independent samples, no satisfactory approach exists for comparing kappas derived from the same or related samples. Resampling methods for comparing kappa coefficients obtained from the same sample are presented Easily undertaken using readily available software, these methods are illustrated by application to a small sample of psychiatric data, as well as to several thousand samples of simulated data. An acceptable type I error rate was exhibited. Resampling techniques — easily implemented and making few assumptions — deserve wider application in psychiatric research.