Natural conversations with a virtual being: How user experience with a current conversational AI model compares to expectations

• Published in: Computer animation and virtual worlds Vol. 34; no. 6
• Main Authors: So, Chaehan, Khvan, Anel, Choi, Wonjun
• Format: Journal Article
• Language: English
• Published: Hoboken, USA John Wiley & Sons, Inc 01.11.2023; Wiley Subscription Services, Inc
• Subjects: approachable; conversational AI; Conversational artificial intelligence; empathetic; engaging; fine‐tuning; Mathematical models; Parameters; Perceptions; Performance measurement; Tuning; User experience; virtual agents
• ISSN: 1546-4261; 1546-427X
• DOI: 10.1002/cav.2149

The present work investigates the effect of natural conversations with virtual beings on user perceptions with a current conversational AI model (Meta's BlenderBot). To this aim, we designed a virtual being from a deep learning‐generated face and a conversational AI model acting as a virtual conversation partner in an online conferencing software and evaluated it in 11 perceptions of social attributes. Compared to prior expectations, participants perceived the virtual being as distinctly higher in warmth (engaging, empathic, and approachable) but lower in realism and credibility after 5 days of 10 min daily conversations (Study 1). Further, we explored the idea of simplifying the technical setup to reduce the technical entry barrier for such AI applications (Study 2). To this aim, we conducted several trials of fine‐tuning a small conversational model of 90 million parameters until its performance metrics improved. Testing this fine‐tuned model with users revealed that this model was not perceived differently from a large conversational model (1.4 billion parameters). In summary, our findings show that recent progress in conversational AI has added warmth‐related aspects to the user experience with virtual beings, and that fine‐tuning a conversational AI model can be effective to reduce technical complexity. We created a virtual being composed of a conversational AI model (run on a cloud‐based GPU) that was visualized by a talking and listening AI‐generated face. We let participants talk to this virtual being in Zoom for five days. Results revealed that Participants evaluated most perceptions of the virtual being as lower than their expectations. Warmth‐related perceptions (approachable, engaging, empathetic) were rated higher than expectations. These three perceptions formed a separate psychological dimension.