Personalized models for facial emotion recognition through transfer learning

• Published in: Multimedia tools and applications Vol. 79; no. 47-48; pp. 35811 - 35828
• Main Authors: Rescigno, Martina, Spezialetti, Matteo, Rossi, Silvia
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.12.2020; Springer Nature B.V
• Subjects: Affective computing; Arousal; Artificial neural networks; Computer Communication Networks; Computer Science; Customization; Data Structures and Information Theory; Datasets; Emotion recognition; Emotions; Feature extraction; Image acquisition; Machine learning; Multimedia Information Systems; Personal information; Sampling methods; Special Purpose and Application-Based Systems; Affective computing; Convolutional neural networks; Transfer learning; Facial emotion recognition
• DOI: 10.1007/s11042-020-09405-4
• ISSN: 1380-7501

Emotions represent a key aspect of human life and behavior. In recent years, automatic recognition of emotions has become an important component in the fields of affective computing and human-machine interaction. Among many physiological and kinematic signals that could be used to recognize emotions, acquiring facial expression images is one of the most natural and inexpensive approaches. The creation of a generalized, inter-subject, model for emotion recognition from facial expression is still a challenge, due to anatomical, cultural and environmental differences. On the other hand, using traditional machine learning approaches to create a subject-customized, personal, model would require a large dataset of labelled samples. For these reasons, in this work, we propose the use of transfer learning to produce subject-specific models for extracting the emotional content of facial images in the valence/arousal dimensions. Transfer learning allows us to reuse the knowledge assimilated from a large multi-subject dataset by a deep-convolutional neural network and employ the feature extraction capability in the single subject scenario. In this way, it is possible to reduce the amount of labelled data necessary to train a personalized model, with respect to relying just on subjective data. Our results suggest that generalized transferred knowledge, in conjunction with a small amount of personal data, is sufficient to obtain high recognition performances and improvement with respect to both a generalized model and personal models. For both valence and arousal dimensions, quite good performances were obtained (RMSE = 0.09 and RMSE = 0.1 for valence and arousal, respectively). Overall results suggested that both the transferred knowledge and the personal data helped in achieving this improvement, even though they alternated in providing the main contribution. Moreover, in this task, we observed that the benefits of transferring knowledge are so remarkable that no specific active or passive sampling techniques are needed for selecting images to be labelled.