Comparing technologies for conveying emotions through realistic avatars in virtual reality‐based metaverse experiences

• Published in: Computer animation and virtual worlds Vol. 34; no. 3-4
• Main Authors: Visconti, Alessandro, Calandra, Davide, Lamberti, Fabrizio
• Format: Journal Article
• Language: English
• Published: Chichester Wiley Subscription Services, Inc 01.05.2023
• Subjects: Algorithms; avatar representation; Avatars; Conveying; Emotions; Eye movements; facial and eye tracking; lip sync approximation; User experience; Virtual environments; Virtual reality
• ISSN: 1546-4261; 1546-427X
• DOI: 10.1002/cav.2188

With the development of metaverse(s), industry and academia are searching for the best ways to represent users' avatars in shared virtual environments (VEs), where real‐time communication between users is required. The expressiveness of avatars is crucial for transmitting emotions that are key for social presence and user experience, and are conveyed via verbal and non‐verbal facial and body signals. In this paper, two real‐time modalities for conveying expressions in virtual reality (VR) via realistic, full‐body avatars are compared by means of a user study. The first modality uses dedicated hardware (i.e., eye and facial trackers) to allow a mapping between the user's facial expressions/eye movements and the avatar model. The second modality relies on an algorithm that, starting from an audio clip, approximates the facial motion by generating plausible lip and eye movements. The participants were requested to observe, for both the modalities, the avatar of an actor performing six scenes involving as many basic emotions. The evaluation considered mainly social presence and emotion conveyance. Results showed a clear superiority of facial tracking when compared to lip sync in conveying sadness and disgust. The same was less evident for happiness and fear. No differences were observed for anger and surprise. Two real‐time modalities for conveying expressions in virtual reality (VR) via realistic, full‐body avatars are compared by means of a user study. The first modality uses dedicated hardware (i.e., eye and facial trackers) and the second modality approximates the facial motion with an algorithm based on audio clips. The participants were requested to observe, for both the modalities, an avatar performing basic emotions, and judge social presence and emotion conveyance.