Dual integrated convolutional neural network for real-time facial expression recognition in the wild

• Published in: The Visual computer Vol. 38; no. 3; pp. 1083 - 1096
• Main Authors: Saurav, Sumeet, Gidde, Prashant, Saini, Ravi, Singh, Sanjay
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.03.2022; Springer Nature B.V
• Subjects: Accuracy; Artificial Intelligence; Artificial neural networks; Computer Graphics; Computer Science; Computer vision; Datasets; Deep learning; Discriminant analysis; Emotions; Face recognition; Image Processing and Computer Vision; Machine learning; Mathematical models; Neural networks; Original Article; Parameters; Pattern recognition; Performance evaluation; Real time; Facial expression recognition; Embedded implementation; Deep convolutional neural network; CNN optimization
• DOI: 10.1007/s00371-021-02069-7
• ISSN: 0178-2789

Automatic recognition of facial expressions in the wild is a challenging problem and has drawn a lot of attention from the computer vision and pattern recognition community. Since their emergence, the deep learning techniques have proved their efficacy in facial expression recognition (FER) tasks. However, these techniques are parameter intensive, and thus, could not be deployed on resource-constrained embedded platforms for real-world applications. To mitigate these limitations of the deep learning inspired FER systems, in this paper, we present an efficient dual integrated convolution neural network (DICNN) model for the recognition of facial expressions in the wild in real-time, running on an embedded platform. The designed DICNN model with just 1.08M parameters and 5.40 MB memory storage size achieves optimal performance by maintaining a proper balance between recognition accuracy and computational efficiency. We evaluated the DICNN model on four FER benchmark datasets (FER2013, FERPlus, RAF-DB, and CKPlus) using different performance evaluation metrics, namely the recognition accuracy, precision, recall, and F1-score. Finally, to provide a portable solution with high throughput inference, we optimized the designed DICNN model using TensorRT SDK and deployed it on an Nvidia Xavier embedded platform. Comparative analysis results with the other state-of-the-art methods revealed the effectiveness of the designed FER system, which achieved competitive accuracy with multi-fold improvement in the execution speed.