Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data

• Published in: Pattern recognition letters Vol. 66; pp. 22 - 30
• Main Authors: Ringeval, Fabien, Eyben, Florian, Kroupi, Eleni, Yuce, Anil, Thiran, Jean-Philippe, Ebrahimi, Touradj, Lalanne, Denis, Schuller, Björn
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 15.11.2015
• Subjects: Audiovisual and physiological data; Context-learning long short-term memory recurrent neural networks; Continuous affect analysis; Multi-task learning; Multimodal fusion; Multitime resolution features extraction; Context-learning long short-term memory recurrent neural networks; Multitime resolution features extraction; Audiovisual and physiological data; Continuous affect analysis; Multi-task learning; Multimodal fusion
• DOI: 10.1016/j.patrec.2014.11.007
• ISSN: 0167-8655

•We study the relevance of context-learning for handling asynchrony of annotation.•We unite audiovisual and physiological data for continuous affect analysis.•We propose multi-time resolution features extraction from multimodal data.•The use of context-learning allows to include reaction time delay of raters.•Fusion of audiovisual and physiological data performs best on arousal and valence. Automatic emotion recognition systems based on supervised machine learning require reliable annotation of affective behaviours to build useful models. Whereas the dimensional approach is getting more and more popular for rating affective behaviours in continuous time domains, e.g., arousal and valence, methodologies to take into account reaction lags of the human raters are still rare. We therefore investigate the relevance of using machine learning algorithms able to integrate contextual information in the modelling, like long short-term memory recurrent neural networks do, to automatically predict emotion from several (asynchronous) raters in continuous time domains, i.e., arousal and valence. Evaluations are performed on the recently proposed RECOLA multimodal database (27 subjects, 5  min of data and six raters for each), which includes audio, video, and physiological (ECG, EDA) data. In fact, studies uniting audiovisual and physiological information are still very rare. Features are extracted with various window sizes for each modality and performance for the automatic emotion prediction is compared for both different architectures of neural networks and fusion approaches (feature-level/decision-level). The results show that: (i) LSTM network can deal with (asynchronous) dependencies found between continuous ratings of emotion with video data, (ii) the prediction of the emotional valence requires longer analysis window than for arousal and (iii) a decision-level fusion leads to better performance than a feature-level fusion. The best performance (concordance correlation coefficient) for the multimodal emotion prediction is 0.804 for arousal and 0.528 for valence.