An Application of Generalizability Theory to Evaluate the Technical Quality of an Alternate Assessment

• Published in: Applied measurement in education Vol. 26; no. 4; pp. 279 - 297
• Main Authors: Taylor, Melinda Ann, Pastor, Dena A.
• Format: Journal Article
• Language: English
• Published: Philadelphia Routledge 01.10.2013; Taylor & Francis Ltd
• Subjects: Accountability; Alternative Assessment; Children & youth; Cognition & reasoning; Disabilities; Error of Measurement; Federal regulation; Generalizability Theory; Grade 10; Grade 5; Grade 8; Learning disabilities; Mathematics Tests; Reading Tests; Scores; Statistical Analysis; Test Construction; Test Reliability; United States > US
• ISSN: 0895-7347; 1532-4818
• DOI: 10.1080/08957347.2013.824450

Although federal regulations require testing students with severe cognitive disabilities, there is little guidance regarding how technical quality should be established. It is known that challenges exist with documentation of the reliability of scores for alternate assessments. Typical measures of reliability do little in modeling multiple sources of error, which are characteristic of alternate assessments. Instead, Generalizability theory (G-theory) allows researchers to identify sources of error and analyze the relative contribution of each source. This study demonstrates an application of G-theory to examine reliability for an alternate assessment. A G-study with the facets rater type, assessment attempts, and tasks was examined to determine the relative contribution of each to observed score variance. Results were used to determine the reliability of scores. The assessment design was modified to examine how changes might impact reliability. As a final step, designs that were deemed satisfactory were evaluated regarding the feasibility of adapting them into a statewide standardized assessment and accountability program.