2-D canonical correlation analysis based image super-resolution scheme for facial emotion recognition

• Published in: Multimedia tools and applications Vol. 81; no. 10; pp. 13911 - 13934
• Main Authors: ullah, Zia, Qi, Lin, Binu, D., Rajakumar, B. R., Mohammed Ismail, B.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.04.2022; Springer Nature B.V
• Subjects: Artificial neural networks; Classification; Computer Communication Networks; Computer Science; Correlation analysis; Data Structures and Information Theory; Emotion recognition; Emotions; Feature extraction; Image acquisition; Image resolution; Machine learning; Multimedia Information Systems; Object recognition; Optimization models; Principal components analysis; Special Purpose and Application-Based Systems; Two dimensional analysis; CNN; 2D CCA; LSTM; Image super-resolution; GM-WLBP; Face image; ITSA; Facial expression
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-022-11922-3

In this research work, a new Image super-resolution-based Face Emotion Recognition Model has been introduced. The proposed work includes two major phases: (a) Facial image super-resolution and (b) Facial emotion recognition. Initially, the collected facial image is subjected to the facial image super-resolution phase, where the Higher Resolution (HR) facial images are subjected to two-dimensional canonical correlation analysis (2D CCA). The acquired HR facial images are considered as the input for facial emotion recognition. From the acquired HR facial images, the face region alone (lips, eyes, and cheeks) is detected by Viola-Jones facial detection model. Subsequently, from the acquired facial regions, the most relevant features like proposed “Geometric Mean based Weighted Local Binary Pattern (GM-WLBP), Gray Level Co-occurrence Matrix ( GLCM )” and generalized low-rank model ( GLRM ) features are extracted. Then, Principal Component Analysis (PCA) technique is applied to solve the curse of dimensionality. Finally, the reduced dimensional features are given to the emotion classification phase to classify the emotions as sad, happy, fear, rage, disgust, and surprise. The proposed hybrid classifier framework includes the renowned Long-Short Term Memory Network (LSTM) and Convolutional Neural Network (CNN) models. These deep learning models is separately trained using the dimensionally reduced features, and the outcomes are combined. Then, the mean value is computed on the final combined outcome (output of LSTM+ output of CNN), which results in the type of emotions. To enhance the classification accuracy, the weight of CNN is fine-tuned by a new Improved Tunicate swarm Optimization Model (ITSA), which is the conceptual improvement of standard Tunicate swarm Optimization (TSA). The performance of the proposed work is evaluated over the existing model to show the supremacy of the proposed work.