Evaluating an Explicit Instruction Teacher Observation Protocol through a Validity Argument Approach

In this study, we examined the scoring and generalizability assumptions of an Explicit Instruction (EI) special education teacher observation protocol using many-faceted Rasch measurement (MFRM). Video observations of classroom instruction from 48 special education teachers across four states were c...

Full description

Saved in:
Bibliographic Details
Published inGrantee Submission
Main Authors Johnson, Evelyn S, Zheng, Yuzhu, Crawford, Angela R, Moylan, Laura A
Format Report
LanguageEnglish
Published 2020
Subjects
Classroom Observation Techniques
Direct Instruction
Generalizability Theory
Interrater Reliability
Item Response Theory
Scoring
Special Education Teachers
Teacher Evaluation
Validity
Online AccessGet full text

Cover

More Information
Summary:In this study, we examined the scoring and generalizability assumptions of an Explicit Instruction (EI) special education teacher observation protocol using many-faceted Rasch measurement (MFRM). Video observations of classroom instruction from 48 special education teachers across four states were collected. External raters (n = 20) were trained to observe and evaluate instruction using the protocol. The results of this study suggest that the scoring rule is appropriate, in that the three-point scale allows for a meaningful way to differentiate various levels of quality of implementation of EI across teachers. Raters consistently scored easier items with higher scores, and more difficult items with a lower score. Additionally, the MFRM results for the rater facet suggest that raters can consistently apply the scoring criteria, and that there is limited rater bias impacting the scores. Implications for research and practice are discussed. [This paper will be published in the "Journal of Experimental Education."]
DOI:10.1080/00220973.2020.1811194