Interrater Reliability for Interdependent Social Network Data: A Generalizability Theory Approach

• Published in: Multivariate behavioral research Vol. 60; no. 3; pp. 444 - 459
• Main Authors: Hove, Debby ten, Jorgensen, Terrence D., van der Ark, L. Andries
• Format: Journal Article
• Language: English
• Published: United States Routledge 04.05.2025
• Subjects: Bayes Theorem; Bayesian hierarchical linear modeling; Computer Simulation; generalizability theory; Humans; Interpersonal Relations; interrater reliability; Markov Chains; Models, Statistical; Monte Carlo Method; Reproducibility of Results; social network data; social relations model; Social Support; interrater reliability; Bayesian hierarchical linear modeling; social relations model; social network data; generalizability theory
• ISSN: 0027-3171; 1532-7906; 1532-7906
• DOI: 10.1080/00273171.2024.2444940

We propose interrater reliability coefficients for observational interdependent social network data, which are dyadic data from a network of interacting subjects that are observed by external raters. Using the social relations model, dyadic scores of subjects' behaviors during these interactions can be decomposed into actor, partner, and relationship effects. These effects constitute different facets of theoretical interest about which researchers formulate research questions. Based on generalizability theory, we extended the social relations model with rater effects, resulting in a model that decomposes the variance of dyadic observational data into effects of actors, partners, relationships, raters, and their statistical interactions. We used the variances of these effects to define intraclass correlation coefficients (ICCs) that indicate the extent the actor, partner, and relationship effects can be generalized across external raters. We proposed Markov chain Monte Carlo estimation of a Bayesian hierarchical linear model to estimate the ICCs, and tested their bias and coverage in a simulation study. The method is illustrated using data on social mimicry.