Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order

• Published in: Pattern recognition Vol. 61; pp. 610 - 628
• Main Authors: Lopes, André Teixeira, de Aguiar, Edilson, De Souza, Alberto F., Oliveira-Santos, Thiago
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.01.2017
• Subjects: Computer vision; Convolutional Neural Networks; Expression specific features; Facial expression recognition; Machine learning; Facial expression recognition; Computer vision; Expression specific features; Convolutional Neural Networks; Machine learning
• DOI: 10.1016/j.patcog.2016.07.026
• ISSN: 0031-3203

Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine learning methods, since people can vary significantly in the way they show their expressions. Even images of the same person in the same facial expression can vary in brightness, background and pose, and these variations are emphasized if considering different subjects (because of variations in shape, ethnicity among others). Although facial expression recognition is very studied in the literature, few works perform fair evaluation avoiding mixing subjects while training and testing the proposed algorithms. Hence, facial expression recognition is still a challenging problem in computer vision. In this work, we propose a simple solution for facial expression recognition that uses a combination of Convolutional Neural Network and specific image pre-processing steps. Convolutional Neural Networks achieve better accuracy with big data. However, there are no publicly available datasets with sufficient data for facial expression recognition with deep architectures. Therefore, to tackle the problem, we apply some pre-processing techniques to extract only expression specific features from a face image and explore the presentation order of the samples during training. The experiments employed to evaluate our technique were carried out using three largely used public databases (CK+, JAFFE and BU-3DFE). A study of the impact of each image pre-processing operation in the accuracy rate is presented. The proposed method: achieves competitive results when compared with other facial expression recognition methods – 96.76% of accuracy in the CK+ database – it is fast to train, and it allows for real time facial expression recognition with standard computers. •A CNN based approach for facial expression recognition.•A set of pre-processing steps allowing for a simpler CNN architecture.•A study of the impact of each pre-processing step in the accuracy.•A study for lowering the impact of the sample presentation order during training.•High facial expression recognition accuracy (96.76%) with real time evaluation.