Joint facial expression recognition and intensity estimation based on weighted votes of image sequences

• Published in: Pattern recognition letters Vol. 92; pp. 25 - 32
• Main Authors: Kamarol, Siti Khairuni Amalina, Jaward, Mohamed Hisham, Kälviäinen, Heikki, Parkkinen, Jussi, Parthiban, Rajendran
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01.06.2017; Elsevier Science Ltd
• Subjects: Change detection; Change-point detection; Complexity; Computer applications; Computer vision; Emotions; Face recognition; Facial expression recognition; Hidden markov models; Image processing systems; Markov analysis; Markov chains; Muscle contraction; Muscles; Object recognition; Pattern recognition; Self expression; Video data; Video-based facial expression intensity analysis; Voting; Voting scheme; Change-point detection; Facial expression recognition; Computer vision; Voting scheme; Video-based facial expression intensity analysis; Hidden markov models
• ISSN: 0167-8655; 1872-7344
• DOI: 10.1016/j.patrec.2017.04.003

•A framework for joint facial expression recognition and intensity estimation from image sequences is proposed.•A feature representation based on weighted votes is also proposed.•Superior performance in estimating facial expression intensities.•Promising results with low computational complexity. Facial behavior consists of dynamically changing properties of facial features as a result of muscle activation. Facial behavior analysis is a challenging problem due to complexity of emotions and variability of the facial expressions associated with the emotions. Most facial expression recognition systems attempt to recognize facial expressions without taking into account the intensity of the expressions. In this paper, a novel framework for facial expression recognition and intensity estimation with low computational complexity requirement is proposed. The algorithm constructs a representation of facial features based on a weighted voting scheme and employs Hidden Markov Models to classify an input video into one of the six basic expressions, namely anger, disgust, fear, happiness, sadness, and surprise. The temporal segments, neutral, onset, and apex, of an expression are then obtained by means of a change-point detector. Evaluations on subject-independent analysis was conducted using Cohn-Kanade dataset and Beihang University facial expression datasets. The proposed approach has demonstrated a superior performance in recognizing facial expressions and estimating expression intensities.