Region Based Parallel Hierarchy Convolutional Neural Network for Automatic Facial Nerve Paralysis Evaluation

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 28; no. 10; pp. 2325 - 2332
• Main Authors: Liu, Xin, Xia, Yifan, Yu, Hui, Dong, Junyu, Jian, Muwei, Pham, Tuan D.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Bells palsy; Diagnostic systems; Evaluation; Face recognition; Facial muscles; Facial nerve; Facial nerve paralysis; Feature extraction; Image sequences; Long short-term memory; LSTM; Muscles; Neural networks; Organs; Paralysis; region of interest; Rehabilitation; severity grade; Shape; spatio-temporal features; Structural hierarchy; Temporal variations
• ISSN: 1534-4320; 1558-0210; 1558-0210
• DOI: 10.1109/TNSRE.2020.3021410

In this article, we propose a parallel hierarchy convolutional neural network (PHCNN) combining a Long Short-Term Memory (LSTM) network structure to quantitatively assess the grading of facial nerve paralysis (FNP) by considering the region-based asymmetric facial features and temporal variation of the image sequences. FNP, such as Bell's palsy, is the most common facial symptom of neuromotor dysfunctions. It causes the weakness of facial muscles for the normal emotional expression and movements. The subjective judgement by clinicians completely depends on individual experience, which may not lead to a uniform evaluation. Existing computer-aided methods mainly rely on some complicated imaging equipment, which is complicated and expensive for facial functional rehabilitation. Compared with the subjective judgment and complex imaging processing, the objective and intelligent measurement can potentially avoid this issue. Considering dynamic variation in both global and regional facial areas, the proposed hierarchical network with LSTM structure can effectively improve the diagnostic accuracy and extract paralysis detail from the low-level shape, contour to sematic level features. By segmenting the facial area into two palsy regions, the proposed method can discriminate FNP from normal face accurately and significantly reduce the effect caused by age wrinkles and unrepresentative organs with shape and position variations on feature learning. Experiment on the YouTube Facial Palsy Database and Extended CohnKanade Database shows that the proposed method is superior to the state of the art deep learning methods.