Human emotion and cognition recognition from body language of the head using soft computing techniques

• Published in: Journal of ambient intelligence and humanized computing Vol. 4; no. 1; pp. 121 - 140
• Main Authors: Zhao, Yisu, Wang, Xin, Goubran, Miriam, Whalen, Thomas, Petriu, Emil M.
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer-Verlag 01.02.2013; Springer Nature B.V
• Subjects: Artificial Intelligence; Cognition; Cognition & reasoning; Computational Intelligence; Culture; Emotion recognition; Emotions; Engineering; Eye movements; Head movement; Language; Original Research; Robotics and Automation; Soft computing; User groups; User Interfaces and Human Computer Interaction; Emotion recognition; Human–computer intelligent interaction; Cognition recognition; Body language of the head; Fuzzy inference systems
• ISSN: 1868-5137; 1868-5145
• DOI: 10.1007/s12652-012-0107-1

To make a computer interface more usable, enjoyable, and effective, it should be able to recognize emotions of its human counterpart. This paper explores new ways to infer the user’s emotions and cognitions from the combination of facial expression (happy, angry, or sad), eye gaze (direct or averted), and head movement (direction and frequency). All of the extracted information is taken as input data and soft computing techniques are applied to infer emotional and cognitional states. The fuzzy rules were defined based on the opinion of an expert in psychology, a pilot group and annotators. Although the creation of the fuzzy rules are specific to a given culture, the idea of integrating the different modalities of the body language of the head is generic enough to be used by any particular target user group from any culture. Experimental results show that this method can be used to successfully recognize 10 different emotions and cognitions.