Fast facial expression recognition using local binary features and shallow neural networks

• Published in: The Visual computer Vol. 36; no. 1; pp. 97 - 112
• Main Authors: Gogić, Ivan, Manhart, Martina, Pandžić, Igor S., Ahlberg, Jörgen
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.01.2020; Springer Nature B.V
• Subjects: Accuracy; Algorithms; Artificial Intelligence; Classification; Computer Graphics; Computer Science; Datasets; Decision trees; Emotions; Face recognition; Image Processing and Computer Vision; Neural networks; Original Article; Facial expression recognition; Decision tree ensembles; Neural networks; Local binary features
• ISSN: 0178-2789; 1432-2315; 1432-2315
• DOI: 10.1007/s00371-018-1585-8

Facial expression recognition applications demand accurate and fast algorithms that can run in real time on platforms with limited computational resources. We propose an algorithm that bridges the gap between precise but slow methods and fast but less precise methods. The algorithm combines gentle boost decision trees and neural networks. The gentle boost decision trees are trained to extract highly discriminative feature vectors (local binary features) for each basic facial expression around distinct facial landmark points. These sparse binary features are concatenated and used to jointly optimize facial expression recognition through a shallow neural network architecture. The joint optimization improves the recognition rates of difficult expressions such as fear and sadness. Furthermore, extensive experiments in both within- and cross-database scenarios have been conducted on relevant benchmark data sets for facial expression recognition: CK+, MMI, JAFFE, and SFEW 2.0. The proposed method (LBF-NN) compares favorably with state-of-the-art algorithms while achieving an order of magnitude improvement in execution time.