It’s Not Only What You Say, But Also How You Say It: Machine Learning Approach to Estimate Trust from Conversation

• Published in: Human factors Vol. 66; no. 6; pp. 1724 - 1741
• Main Authors: Li, Mengyao, Erickson, Isabel M, Cross, Ernest V, Lee, John D
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01.06.2024; Human Factors and Ergonomics Society
• Subjects: Acoustics; Adult; Algorithms; Communication; Decision making; Decision trees; Female; Human-Computer Interaction, Computer Systems; Humans; Learning algorithms; Machine Learning; Male; Mental task performance; Space exploration; Trust; Trusting automation; trust measurement; model visualization and explainability; machine learning; human-AI-robot teaming
• ISSN: 0018-7208; 1547-8181; 1547-8181
• DOI: 10.1177/00187208231166624

Objective The objective of this study was to estimate trust from conversations using both lexical and acoustic data. Background As NASA moves to long-duration space exploration operations, the increasing need for cooperation between humans and virtual agents requires real-time trust estimation by virtual agents. Measuring trust through conversation is a novel and unintrusive approach. Method A 2 (reliability) × 2 (cycles) × 3 (events) within-subject study with habitat system maintenance was designed to elicit various levels of trust in a conversational agent. Participants had trust-related conversations with the conversational agent at the end of each decision-making task. To estimate trust, subjective trust ratings were predicted using machine learning models trained on three types of conversational features (i.e., lexical, acoustic, and combined). After training, model explanation was performed using variable importance and partial dependence plots. Results Results showed that a random forest algorithm, trained using the combined lexical and acoustic features, predicted trust in the conversational agent most accurately ( R a d j 2 = 0.71 ) . The most important predictors were a combination of lexical and acoustic cues: average sentiment considering valence shifters, the mean of formants, and Mel-frequency cepstral coefficients (MFCC). These conversational features were identified as partial mediators predicting people’s trust. Conclusion Precise trust estimation from conversation requires lexical cues and acoustic cues. Application These results showed the possibility of using conversational data to measure trust, and potentially other dynamic mental states, unobtrusively and dynamically.