Multi angle optimal pattern-based deep learning for automatic facial expression recognition

• Published in: Pattern recognition letters Vol. 139; pp. 157 - 165
• Main Authors: Jain, Deepak Kumar, Zhang, Zhang, Huang, Kaiqi
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01.11.2020; Elsevier Science Ltd
• Subjects: Algorithms; Alignment; Artificial neural networks; CNN; Deep learning; Feature extraction; Illumination; Image analysis; Image processing; Long short-term memory; LSTM; Machine learning; Neural networks; Occupancy; Optimization; Pattern recognition; STM; Subtraction; SURF; Texture; Variation; SURF; CNN; LSTM; 41A10; 65D05; 65D17; STM
• ISSN: 0167-8655; 1872-7344
• DOI: 10.1016/j.patrec.2017.06.025

•Rectification of problems due to sudden illumination changes and proper alignment of feature set for an effective FER.•Isolation of foreground from the background of an image to facilitate the facial key points extraction.•Elimination of unwanted noise and smoothen by using AMF filtering technique.•MATP-based texture pattern and DESURF-LOP-based facial key points to the FER outperform the traditional methods.•PPCSO-based feature selection and LSTM-CNN based classification recognize the expressions accurately. Facial Expression Recognition (FER) plays the vital role in the Human Computer Interface (HCI) applications. The illumination and pose variations affect the FER adversely. The projection of complex 3D actions on the image plane and the inaccurate alignment are the major issues in the FER process. This paper presents the novel Multi-Angle Optimal Pattern-based Deep Learning (MAOP-DL) method to rectify the problem from sudden illumination changes, find the proper alignment of a feature set by using multi-angle-based optimal configurations. The proposed method includes the five major processes as Extended Boundary Background Subtraction (EBBS), Multi-Angle Texture Pattern+STM, Densely Extracted SURF+Local Occupancy Pattern (LOP), Priority Particle Cuckoo Search Optimization (PPCSO) and Long Short-Term Memory -Convolutional Neural Network (LSTM-CNN). Initially, the EBBS algorithm subtracts the background and isolates the foreground from the images which overcome the illumination and pose variation. Then, the MATP-STM extracts the texture patterns and DESURF-LOP extracts the relevant key features of the facial points. The PPCSO algorithm selects the relevant features from the MATP-STM feature set to speed up the classification. The employment of LSTM-CNN predicts the required label for the facial expressions.The major key findings of the proposed work are clear image analysis, effective handling of pose/illumination variations and the facial alignment. The proposed MAOP-DL validates its effectiveness on two standard databases such as CK+ and MMI regarding various metrics and confirm their assurance of wide applicability in recent applications.