Cross-Language Speech Emotion Recognition Using Bag-of-Word Representations, Domain Adaptation, and Data Augmentation

• Published in: Sensors (Basel, Switzerland) Vol. 22; no. 17; p. 6445
• Main Authors: Kshirsagar, Shruti, Falk, Tiago H.
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 26.08.2022; MDPI
• Subjects: Accuracy; Adaptation; Arousal; bag of audio words; cross-language speech emotion recognition; Data augmentation; Data mining; Datasets; domain adaptation; Domains; Emotion recognition; Emotions; Experiments; Language; Languages; Machine learning; Methods; modulation spectrum; Multilingualism; Neural networks; Speech; Speech recognition; Hungary; France; Germany
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s22176445

To date, several methods have been explored for the challenging task of cross-language speech emotion recognition, including the bag-of-words (BoW) methodology for feature processing, domain adaptation for feature distribution “normalization”, and data augmentation to make machine learning algorithms more robust across testing conditions. Their combined use, however, has yet to be explored. In this paper, we aim to fill this gap and compare the benefits achieved by combining different domain adaptation strategies with the BoW method, as well as with data augmentation. Moreover, while domain adaptation strategies, such as the correlation alignment (CORAL) method, require knowledge of the test data language, we propose a variant that we term N-CORAL, in which test languages (in our case, Chinese) are mapped to a common distribution in an unsupervised manner. Experiments with German, French, and Hungarian language datasets were performed, and the proposed N-CORAL method, combined with BoW and data augmentation, was shown to achieve the best arousal and valence prediction accuracy, highlighting the usefulness of the proposed method for “in the wild” speech emotion recognition. In fact, N-CORAL combined with BoW was shown to provide robustness across languages, whereas data augmentation provided additional robustness against cross-corpus nuance factors.