Applying Self-Supervised Representation Learning for Emotion Recognition Using Physiological Signals

• Published in: Sensors (Basel, Switzerland) Vol. 22; no. 23; p. 9102
• Main Authors: Montero Quispe, Kevin G, Utyiama, Daniel M S, Dos Santos, Eulanda M, Oliveira, Horácio A B F, Souto, Eduardo J P
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 23.11.2022; MDPI
• Subjects: Affect (Psychology); Affective computing; Algorithms; Analysis; Annotations; Behavior; Electrocardiography; Emotion recognition; Emotions; Emotions - physiology; Humans; Machine Learning; Medical equipment; Nervous system; Neural networks; Neural Networks, Computer; Pattern recognition systems; physiological signals; Physiology; Recognition, Psychology; representation learning; Representations; self-supervised learning; User experience; Wearable computers; wearable sensors; self-supervised learning; wearable sensors; physiological signals; representation learning; emotion recognition
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s22239102

The use of machine learning (ML) techniques in affective computing applications focuses on improving the user experience in emotion recognition. The collection of input data (e.g., physiological signals), together with expert annotations are part of the established standard supervised learning methodology used to train human emotion recognition models. However, these models generally require large amounts of labeled data, which is expensive and impractical in the healthcare context, in which data annotation requires even more expert knowledge. To address this problem, this paper explores the use of the self-supervised learning (SSL) paradigm in the development of emotion recognition methods. This approach makes it possible to learn representations directly from unlabeled signals and subsequently use them to classify affective states. This paper presents the key concepts of emotions and how SSL methods can be applied to recognize affective states. We experimentally analyze and compare self-supervised and fully supervised training of a convolutional neural network designed to recognize emotions. The experimental results using three emotion datasets demonstrate that self-supervised representations can learn widely useful features that improve data efficiency, are widely transferable, are competitive when compared to their fully supervised counterparts, and do not require the data to be labeled for learning.