Interrater Agreement Between Visual Analysts of Single-Case Data A Meta-Analysis

• Published in: Behavior modification Vol. 39; no. 4; pp. 510 - 541
• Main Authors: Ninci, Jennifer, Vannest, Kimberly J., Willson, Victor, Zhang, Nan
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01.07.2015; SAGE PUBLICATIONS, INC
• Subjects: Behavior; Data Interpretation, Statistical; Effect Modifier, Epidemiologic; Humans; Meta-analysis; Observer Variation; Pattern Recognition, Visual; Reproducibility of Results; Systematic review; Variables; Visualization; agreement; visual analysis; single-case research; reliability
• ISSN: 0145-4455; 1552-4167
• DOI: 10.1177/0145445515581327

Visual analysis is the most widely applied method of data interpretation for single-case research as it encompasses multifaceted considerations relevant to evaluating behavior change. However, a previous research synthesis found low levels of interrater agreement between visually analyzed ratings of graphed data across all variables under analysis. The purpose of this meta-analysis was to evaluate the peer-reviewed literature to date for potential moderators affecting the proportion of interrater agreement between visual analysts. Nineteen articles with 32 effects were assembled. Potential moderators evaluated included (a) design families, (b) rater expertise, (c) the provision of contextual information for graphs, (d) the use of visual aids, (e) the provision of an operational definition of the construct being rated, and (f) rating scale ranges. Results yielded an overall weighted interrater agreement proportion of .76. Moderator variables identified produced low to adequate levels of interrater agreement. Practical recommendations for future research are discussed.