A Preliminary Investigation of the Empirical Validity of Study Quality Appraisal

• Published in: Journal of learning disabilities Vol. 50; no. 1; pp. 14 - 22
• Main Authors: Cook, Bryan G., Dupuis, Danielle N., Jitendra, Asha K.
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01.01.2017; SAGE Publications and Hammill Institute on Disabilities; SAGE PUBLICATIONS, INC
• Subjects: Appraisal; Bias; Data Analysis; Educational Measurement - standards; Educational Practices; Educational Quality; Educational Researchers; Evaluation Studies as Topic; Evidence based; Evidence Based Practice; Experimental Programs; Experimenter Characteristics; Humans; Instructional Effectiveness; Intervention; Investigations; Learning Disabilities; Learning disabled students; Learning Disorders; Maximum Likelihood Statistics; Meta Analysis; Outcome Measures; Participant Characteristics; Quality; Quality Assurance; Reproducibility of Results; Research Methodology; Research Reports; Review Literature as Topic; Special Education; Studies; Teaching Methods; Test Reliability; Test Validity; treatment; research method; quantitative; efficacy
• ISSN: 0022-2194; 1538-4780
• DOI: 10.1177/0022219415581178

When classifying the evidence base of practices, special education scholars typically appraise study quality to identify and exclude from consideration in their reviews unacceptable-quality studies that are likely biased and might bias review findings if included. However, study quality appraisals used in the process of identifying evidence-based practices for students with learning and other disabilities have not been empirically validated (e.g., studies classified as unacceptable quality shown to have different, and presumably more biased, effects than high-quality studies). Using Gersten et al.’s (2005) approach for appraising the quality of group experimental studies in special education, we examined whether (a) studies classified as unacceptable quality and high quality had meaningfully different effects and (b) unacceptable-quality studies were more likely to have outlying effects than high-quality studies among 36 group experimental studies that investigated the effectiveness of instructional practices for students with learning disabilities. Our preliminary analyses found that the effects of unacceptable-quality studies were not meaningfully different from the effects of high-quality studies. We discuss implications of these findings and call for more research to be conducted in this area.