Building an ICCN Multimodal Classifier of Aggressive Political Debate Style: Towards a Computational Understanding of Candidate Performance Over Time

• Published in: Communication methods and measures Vol. 18; no. 1; pp. 30 - 47
• Main Authors: Shah, Dhavan V., Sun, Zhongkai, Bucy, Erik P., Kim, Sang Jung, Sun, Yibing, Li, Mengyu, Sethares, William
• Format: Journal Article
• Language: English
• Published: Philadelphia Routledge 02.01.2024; Taylor & Francis Ltd
• Subjects: Aggression; Aggressiveness; Classifiers; Communication; Computer vision; Debates; Dominance displays; Interaction Canonical Correlation Network; multimodal classification; political performance; presidential debates; Segments; Social research; Speech; Trump
• ISSN: 1931-2458; 1931-2466
• DOI: 10.1080/19312458.2023.2227093

Understanding the implications of aggressive political debate style amid corrosive modes of campaign politics requires fine-grained analyses of political performance, attending to multiple communication modalities. Politicians' facial expressions, emotional tone, and speech content can all independently convey aggression and dominance, and often work in combination for purposes of emphasis. Yet micro-coding individual visual, tonal, and verbal features across more than a handful of debate segments becomes extremely labor intensive, hampering research, especially historical, longitudinal, and cross-cultural work. To address this limitation, we develop a novel multimodal classifier using an Interaction Canonical Correlation Network (ICCN) that incorporates video and audio features with speech coding of candidate debate performance, trained on a 20% sample of 10-second segments from each of the first televised U.S presidential debates between 1980 and 2020. In the analysis, we demonstrate this classifier can accurately detect aggression by political candidates in U.S. debates. We sharpen its performance by distinguishing between debate eras characterized by lower and higher levels of aggression and validate the approach by comparing the performance of unimodal with multimodal classification. This classifier opens new avenues for computational social science research, including explaining candidate behavior within debates at a larger scale and across different eras.