Is Interactional Dissynchrony a Clue to Deception? Insights From Automated Analysis of Nonverbal Visual Cues

• Published in: IEEE transactions on cybernetics Vol. 45; no. 3; pp. 492 - 506
• Main Authors: Xiang Yu, Shaoting Zhang, Zhennan Yan, Fei Yang, Junzhou Huang, Dunbar, Norah E., Jensen, Matthew L., Burgoon, Judee K., Metaxas, Dimitris N.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2015; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Deception; Deception detection; Educational institutions; expression recognition; Face; Face - physiology; Face recognition; face tracking; Facial Expression; Feature extraction; gesture detection; Humans; Image Processing, Computer-Assisted - methods; interactional synchrony; Pattern Recognition, Automated - methods; Shape; Tracking; Deception detection; face tracking; gesture detection; expression recognition; interactional synchrony
• ISSN: 2168-2267; 2168-2275
• DOI: 10.1109/TCYB.2014.2329673

Detecting deception in interpersonal dialog is challenging since deceivers take advantage of the give-and-take of interaction to adapt to any sign of skepticism in an interlocutor's verbal and nonverbal feedback. Human detection accuracy is poor, often with no better than chance performance. In this investigation, we consider whether automated methods can produce better results and if emphasizing the possible disruption in interactional synchrony can signal whether an interactant is truthful or deceptive. We propose a data-driven and unobtrusive framework using visual cues that consists of face tracking, head movement detection, facial expression recognition, and interactional synchrony estimation. Analysis were conducted on 242 video samples from an experiment in which deceivers and truth-tellers interacted with professional interviewers either face-to-face or through computer mediation. Results revealed that the framework is able to automatically track head movements and expressions of both interlocutors to extract normalized meaningful synchrony features and to learn classification models for deception recognition. Further experiments show that these features reliably capture interactional synchrony and efficiently discriminate deception from truth.