The modeling of human facial pain intensity based on Temporal Convolutional Networks trained with video frames in HSV color space

• Published in: Applied soft computing Vol. 97; p. 106805
• Main Authors: Bargshady, Ghazal, Zhou, Xujuan, Deo, Ravinesh C., Soar, Jeffrey, Whittaker, Frank, Wang, Hua
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.12.2020
• Subjects: Facial expression; HSV color space; Pain detection; Temporal convolutional network; Video analysis; Temporal convolutional network; Video analysis; Facial expression; Pain detection; HSV color space
• ISSN: 1568-4946; 1872-9681
• DOI: 10.1016/j.asoc.2020.106805

An accurate detection and management of pain, measured through its relative intensity, plays an important role in the treatment of disease and reducing a patient’s discomfort. As it is relatively difficult to assess, describe, evaluate and manage the pain level using a patient’s self-report, automated pain-detecting tools can provide useful information to assist in the management of pain intensity. This study proposes a new predictive modeling framework that employs a modified Temporal Convolutional Network (TCN) algorithm to recognize the pain intensity prevalent in patients’ video frames collected as part of UNBC-McMaster Shoulder Pain Archive and MIntPAIN databases. The inputs of the proposed TCN network is composed of the extracted and reduced face image features from a fine-tuned VGG-Face and principal component analysis (PCA) with Hue, Saturation, Value (HSV) color spaces video images. The results of TCN based predictive model, employing a long short-term memory (LSTM) model as well as other state-of-the art models, show that the proposed approach performs faster with a high level of efficiency. This is demonstrated by the low magnitude of error metrics (i.e., Mean Squared Error = 0.0629, Mean Absolute Error = 0.1021, correctness validation results represented by Area under Curve = 85% and accuracy metric = 92.44%). Considering the efficiency of the proposed TCN framework, integrating fine-tuned VGG-Face and PCA with Hue, Saturation, Value (HSV) color spaces video images for pain intensity estimation, the present study affirms that the new method can be adopted as an automatic health informatics tool, mainly for pain detection, and subsequently, implemented in the pain management area. •Temporal Convolutional Network, TCN predictive model for pain modelling is proposed.•The TCN model considers face images in Hue, Saturation, Value (HSV) colour space.•Fine-tuned VGGFace is incorporated in TCN algorithm to extract face image features.•Predictive model is tested against UNBC-McMaster Shoulder Pain & MIntPAIN database.•The proposed TCN framework has significant implications in health informatics.