Automatic action unit detection in infants using convolutional neural network

• Published in: International Conference on Affective Computing and Intelligent Interaction and workshops Vol. 2017; pp. 216 - 221
• Main Authors: Hammal, Zakia, Chu, Wen-Sheng, Cohn, Jeffrey F., Heike, Carrie, Speltz, Matthew L.
• Format: Conference Proceeding Journal Article
• Language: English
• Published: United States IEEE 01.10.2017
• Subjects: Encoding; Face; Gold; Lips; Manuals; Pediatrics; Task analysis
• ISSN: 2156-8103; 2156-8111
• DOI: 10.1109/ACII.2017.8273603

Action unit detection in infants relative to adults presents unique challenges. Jaw contour is less distinct, facial texture is reduced, and rapid and unusual facial movements are common. To detect facial action units in spontaneous behavior of infants, we propose a multi-label Convolutional Neural Network (CNN). Eighty-six infants were recorded during tasks intended to elicit enjoyment and frustration. Using an extension of FACS for infants (Baby FACS), over 230,000 frames were manually coded for ground truth. To control for chance agreement, inter-observer agreement between Baby-FACS coders was quantified using free-margin kappa. Kappa coefficients ranged from 0.79 to 0.93, which represents high agreement. The multi-label CNN achieved comparable agreement with manual coding. Kappa ranged from 0.69 to 0.93. Importantly, the CNN-based AU detection revealed the same change in findings with respect to infant expressiveness between tasks. While further research is needed, these findings suggest that automatic AU detection in infants is a viable alternative to manual coding of infant facial expression.