A Comprehensive Multimodal Humanoid System for Personality Assessment Based on the Big Five Model

• Published in: IEEE access Vol. 12; pp. 84261 - 84272
• Main Authors: Jaffar, Anum, Ali, Sara, Fahad Iqbal, Khawaja, Ayaz, Yasar, Ansari, Ali R., Fayyaz, Muhammad A. B., Nawaz, Raheel
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Artificial neural networks; Behavioral sciences; Big-five model; Convolutional neural networks; Data mining; Emotion recognition; Facial animation; Head; Head movement; Human engineering; Human-robot interaction; Humanoid; Humanoid robots; Multimodal sensors; non-verbal cues; Personality; personality prediction; Personality traits; Pose estimation; Predictive models; Questionnaires; Robots; Sentiment analysis; Speech recognition; Training
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2024.3412931

Personality analysis allows the experts to get insights into an individual's conduct, vulnerabilities, and prospective capabilities. Some common methods employed for personality prediction include text analysis, social media data, facial expressions, and emotional speech extraction. Recently, some studies have utilized the big five model to predict personality traits using non-verbal cues (gaze score, body motion, head motion). However, these studies mostly target only three aspects of the big five mode. None of the studies so far have used non-verbal cues to target all five traits (extraversion, openness, neuroticism, agreeableness, and conscientiousness) of the Big Five model. In this paper, we propose a multi-modal system that predicts all five personality traits of the Big Five model using non-verbal cues (facial expressions, head poses, body poses), 44-item Big Five Inventory (BFI) questionnaire, and expert analysis. The facial expression module utilizes the Face Emotion Recognition Plus (FER+) dataset trained with Convolution Neural Network (CNN) model achieving 95.14% accuracy. Evaluating 16 subjects in verbal interaction with humanoid robot NAO, we combined questionnaire feedback, human-robot interaction data, and expert perspectives to deduce their Big Five traits. Findings reveal 100% accuracy in personality prediction via expert insights and the system, and 75% for the questionnaire-based approach.