Estimating with a Latent Class Model the Reliability of Nominal Judgments Upon Which Two Raters Agree

Because nominal-scale judgments cannot directly be aggregated into meaningful composites, the addition of a second rater is usually motivated by a desire to estimate the quality of a single rater's classifications rather than to improve reliability. When raters agree, the aggregation problem do...

Full description

Saved in:
Bibliographic Details
Published inEducational and psychological measurement Vol. 66; no. 5; pp. 739 - 747
Main Authors Schuster, Christof, Smith, David A.
Format Journal Article
LanguageEnglish
Published Thousand Oaks, CA SAGE Publications 01.10.2006
Sage
SAGE PUBLICATIONS, INC
Subjects
Biological and medical sciences
Estimating techniques
Evaluation Criteria
Fundamental and applied biological sciences. Psychology
Interrater Reliability
Measurement techniques
Measures (Individuals)
Psychology. Psychoanalysis. Psychiatry
Psychology. Psychophysiology
Psychometrics. Statistics. Methodology
Ratings & rankings
Reliability
Statistics. Mathematics
Tests
latent class analysis
nominal judgments
rater reliability
rater agreement
Human
Statistical analysis
Psychometrics
Statistical estimation
Test reliability
Statistical method
Interrater agreement
Latent class model
Judgment
Online AccessGet full text

Cover

More Information
Summary:Because nominal-scale judgments cannot directly be aggregated into meaningful composites, the addition of a second rater is usually motivated by a desire to estimate the quality of a single rater's classifications rather than to improve reliability. When raters agree, the aggregation problem does not arise. Nevertheless, a proportion of this agreement is likely to have occurred by chance, so reliability issues remain even if the focus is on nominal judgments upon which raters concur. In this article, the reliability of nominal judgments upon which raters agree is addressed in the framework of a latent class model that includes a systematic agreement parameter. It is shown that if attention is limited to agreement cases, the value of this parameter increases as expected. Circumstances under which agreement case reliability is important are discussed.
ISSN:0013-1644
1552-3888
DOI:10.1177/0013164405285905