Unsupervised Multimodal Learning for Dependency-Free Personality Recognition

• Published in: IEEE transactions on affective computing Vol. 15; no. 3; pp. 1053 - 1066
• Main Authors: Ghassemi, Sina, Zhang, Tianyi, van Breda, Ward, Koutsoumpis, Antonis, Oostrom, Janneke K., Holtrop, Djurre, de Vries, Reinout E.
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.07.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Annotations; Audio data; Datasets; Deep learning; Emotion recognition; Feature extraction; feature fusion; Feature recognition; generalization performance; Machine learning; multimedia signal processing; multimodal systems; Personality; Personality assessment; Personality traits; Prediction algorithms; Testing; Time domain analysis; Training; transfer learning; Unsupervised learning; Visual tasks; Visualization
• ISSN: 1949-3045; 1949-3045
• DOI: 10.1109/TAFFC.2023.3318367

Recent advances in AI-based learning models have significantly increased the accuracy of Automatic Personality Recognition (APR). However, these methods either require training data from the same subject or the meta-information from the training set to learn the personality-related features (i.e., subject-dependency). The variance of feature extraction for different subjects compromises the possibility of designing a dependency-free system for APR. To address this problem, we present an unsupervised multimodal learning framework to infer personality traits from audio, visual, and verbal modalities. Our method both extracts the handcraft features and transfers deep-learning based embeddings from other tasks (e.g., emotion recognition) to recognize personality traits. Since these representations are extracted locally in the time domain, we present an unsupervised temporal aggregation method to aggregate the extracted features over the temporal dimension. We evaluate our method on the ChaLearn dataset, the most widely referenced dataset for APR, using a dependency-free split of the dataset. Our results show that the proposed feature extraction and temporal aggregation modules do not require personality annotations in training but still outperform other state-of-the-art baseline methods. We also address the problem of subject-dependency in the original split of the ChaLearn dataset. The newly proposed split (i.e., data for training, validation, and testing) of the dataset can benefit the community by providing a more accurate method to validate the subject-generalizability of APR algorithms.