Data quality and reliability metrics for event-related potentials (ERPs): The utility of subject-level reliability

• Published in: International journal of psychophysiology Vol. 165; pp. 121 - 136
• Main Authors: Clayson, Peter E., Brush, C.J., Hajcak, Greg
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 01.07.2021
• Subjects: Data quality; Error-related negativity (ERN); Event-related potentials (ERPs); Generalizability theory; Multilevel modeling; Psychometric reliability; Reward positivity (RewP); Error-related negativity (ERN); Multilevel modeling; Event-related potentials (ERPs); Reward positivity (RewP); Psychometric reliability; Data quality; Generalizability theory
• ISSN: 0167-8760; 1872-7697; 1872-7697
• DOI: 10.1016/j.ijpsycho.2021.04.004

Event-related brain potentials (ERPs) represent direct measures of neural activity that are leveraged to understand cognitive, affective, sensory, and motor processes. Every ERP researcher encounters the obstacle of determining whether measurements are precise or psychometrically reliable enough for an intended purpose. In this primer, we review three types of measurements metrics: data quality, group-level internal consistency, and subject-level internal consistency. Data quality estimates characterize the precision of ERP scores but provide no inherent information about whether scores are precise enough for examining individual differences. Group-level internal consistency characterizes the ratio of between-person differences to the precision of those scores, and provides a single internal consistency estimate for an entire group of participants that risks masking low internal consistency for some individuals. Subject-level internal consistency considers the precision of an ERP score for a person relative to between-person differences for a group, and an estimate is yielded for each individual. We apply each metric to published error-related negativity (ERN) and reward positivity (RewP) data and demonstrate how failing to consider data quality and internal consistency can undermine statistical inferences. We conclude with general comments on how these estimates may be used to improve measurement quality and methodological transparency. Subject-level internal consistency computation is implemented within the ERP Reliability Analysis (ERA) Toolbox. •Psychometric reliability is critical for studies of individual differences.•Group-level reliability estimates risk masking low reliability.•Subject-level estimates consider the precision of scores for a person.•Data quality and subject-level estimates can improve measurement quality.•Subject-level reliability is implemented within the ERA Toolbox.