Examining the normality assumption of a design-comparable effect size in single-case designs

• Published in: Behavior research methods Vol. 56; no. 1; pp. 379 - 405
• Main Authors: Chen, Li-Ting, Chen, Yi-Kai, Yang, Tong-Rong, Chiang, Yu-Shan, Hsieh, Cheng-Yu, Cheng, Che, Ding, Qi-Wen, Wu, Po-Ju, Peng, Chao-Ying Joanne
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.01.2024; Springer Nature B.V
• Subjects: Behavioral Science and Psychology; Cognitive Psychology; Meta-analysis; Psychology; Intervention; Design comparable; Normality; Single-case; Standardized mean difference; Effect size
• ISSN: 1554-3528; 1554-3528
• DOI: 10.3758/s13428-022-02035-8

What Works Clearinghouse (WWC, 2022 ) recommends a design-comparable effect size (D-CES; i.e., g AB ) to gauge an intervention in single-case experimental design (SCED) studies, or to synthesize findings in meta-analysis. So far, no research has examined g AB ’s performance under non-normal distributions. This study expanded Pustejovsky et al. ( 2014 ) to investigate the impact of data distributions, number of cases ( m ), number of measurements ( N ), within-case reliability or intra-class correlation (ρ), ratio of variance components (λ), and autocorrelation (ϕ) on g AB in multiple-baseline (MB) design. The performance of g AB was assessed by relative bias ( RB ), relative bias of variance ( RBV ), MSE , and coverage rate of 95% CIs ( CR ). Findings revealed that g AB was unbiased even under non-normal distributions. g AB ’s variance was generally overestimated, and its 95% CI was over-covered, especially when distributions were normal or nearly normal combined with small m and N . Large imprecision of g AB occurred when m was small and ρ was large. According to the ANOVA results, data distributions contributed to approximately 49% of variance in RB and 25% of variance in both RBV and CR . m and ρ each contributed to 34% of variance in MSE . We recommend g AB for MB studies and meta-analysis with N  ≥ 16 and when either (1) data distributions are normal or nearly normal, m  = 6, and ρ = 0.6 or 0.8, or (2) data distributions are mildly or moderately non-normal, m  ≥ 4, and ρ = 0.2, 0.4, or 0.6. The paper concludes with a discussion of g AB ’s applicability and design-comparability, and sound reporting practices of ES indices.