Generalizability theory for the perplexed: A practical introduction and guide: AMEE Guide No. 68

• Published in: Medical teacher Vol. 34; no. 11; pp. 960 - 992
• Main Authors: Bloch, Ralph, Norman, Geoffrey
• Format: Journal Article
• Language: English
• Published: England Informa UK, Ltd 01.11.2012; Taylor & Francis; Taylor & Francis Ltd
• Subjects: Data Interpretation, Statistical; Educational Measurement - methods; Educational Practices; Error of Measurement; Generalizability; Generalizability Theory; Generalizations; Guidelines as Topic; Humans; Judgment; Mathematics; Measurement; Personality tests; Physicians; Psychological tests; Psychometric tests; Psychometrics; Rating Scales; Reliability; Reproducibility of Results; Research Design; Statistical Analysis
• ISSN: 0142-159X; 1466-187X; 1466-187X
• DOI: 10.3109/0142159X.2012.703791

Background: Generalizability theory (G theory) is a statistical method to analyze the results of psychometric tests, such as tests of performance like the Objective Structured Clinical Examination, written or computer-based knowledge tests, rating scales, or self-assessment and personality tests. It is a generalization of classical reliability theory, which examines the relative contribution of the primary variable of interest, the performance of subjects, compared to error variance. In G theory, various sources of error contributing to the inaccuracy of measurement are explored. G theory is a valuable tool in judging the methodological quality of an assessment method and improving its precision. Aim: Starting from basic statistical principles, we gradually develop and explain the method. We introduce tools to perform generalizability analysis, and illustrate the use of generalizability analysis with a series of common, practical examples in educational practice. Conclusion: We realize that statistics and mathematics can be either boring or fearsome to many physicians and educators, yet we believe that some foundations are necessary for a better understanding of generalizability analysis. Consequently, we have tried, wherever possible, to keep the use of equations to a minimum and to use a conversational and slightly "off-serious" style.