Path Modeling of Visual Attention, User Perceptions, and Behavior Change Intentions in Conversations With Embodied Agents in VR

• Published in: Computer animation and virtual worlds Vol. 36; no. 3
• Main Authors: Vankit, Sagar A., Genaro Motti, Vivian, Do, Tiffany D., Zamanifard, Samaneh, Diaz, Deyrel, Duchowski, Andrew T., Knijnenburg, Bart P., Volonte, Matias
• Format: Journal Article
• Language: English
• Published: Hoboken, USA John Wiley & Sons, Inc 01.05.2025; Wiley Subscription Services, Inc
• Subjects: embodied conversational agents; Factorial design; multimodal conversations; Subtitles & subtitling; Three dimensional models; Virtual reality
• DOI: 10.1002/cav.70028
• ISSN: 1546-4261

ABSTRACT This study examines how subtitles and image visualizations influence gaze behavior, working alliance, and behavior change intentions in virtual health conversations with ECAs. Visualizations refer to images on a 3D model TV and text on a virtual whiteboard, both reinforcing key content conveyed by the ECA. Using a 2 ×$$ \times $$ 2 factorial design, participants were randomly assigned to one of four conditions: no subtitles or visualizations (Control), subtitles only (SUB), visualizations only (VIS), or both subtitles and visualizations (VISSUB). Structural equation path modeling showed that SUB and VIS individually reduced gaze toward the ECA, whereas VISSUB moderated this reduction, resulting in less gaze loss than the sum of either condition alone. Gaze behavior was positively associated with working alliance, and perceptions of enjoyment and appropriateness influenced engagement, which in turn predicted behavior change intentions. VIS was negatively associated with behavior change intentions, suggesting that excessive visual input may introduce cognitive trade‐offs. This study investigates how multimodal design features like subtitles and visualizations affect user attention, perceptions, gaze behavior and motivation in virtual health conversations with Embodied Conversational Agents (ECAs). Conducted in immersive VR, participants interacted with a virtual health coach delivering guidance on healthy eating. Results from path modeling revealed that visual attention to the ECA was linked to stronger working alliance and behavior change intentions, while excessive visual input introduced trade‐offs in engagement and persuasion.