Automatic rating of therapist facilitative interpersonal skills in text: A natural language processing application

• Published in: Frontiers in digital health Vol. 4; p. 917918
• Main Authors: Zech, James M., Steele, Robert, Foley, Victoria K., Hull, Thomas D.
• Format: Journal Article
• Language: English
• Published: Frontiers Media S.A 16.08.2022
• Subjects: BERT; common factors; Digital Health; Facilitative interpersonal skills; machine learning; messaging therapy
• ISSN: 2673-253X; 2673-253X
• DOI: 10.3389/fdgth.2022.917918

Background While message-based therapy has been shown to be effective in treating a range of mood disorders, it is critical to ensure that providers are meeting a consistently high standard of care over this medium. One recently developed measure of messaging quality–The Facilitative Interpersonal Skills Task for Text (FIS-T)–provides estimates of therapists’ demonstrated ability to convey psychotherapy's common factors (e.g., hopefulness, warmth, persuasiveness) over text. However, the FIS-T's scoring procedure relies on trained human coders to manually code responses, thereby rendering the FIS-T an unscalable quality control tool for large messaging therapy platforms. Objective In the present study, researchers developed two algorithms to automatically score therapist performance on the FIS-T task. Methods The FIS-T was administered to 978 messaging therapists, whose responses were then manually scored by a trained team of raters. Two machine learning algorithms were then trained on task-taker messages and coder scores: a support vector regressor (SVR) and a transformer-based neural network (DistilBERT). Results The DistilBERT model had superior performance on the prediction task while providing a distribution of ratings that was more closely aligned with those of human raters, versus SVR. Specifically, the DistilBERT model was able to explain 58.8% of the variance ( R 2 = 0.588) in human-derived ratings and realized a prediction mean absolute error of 0.134 on a 1–5 scale. Conclusions Algorithms can be effectively used to ensure that digital providers meet a consistently high standard of interactions in the course of messaging therapy. Natural language processing can be applied to develop new quality assurance systems in message-based digital psychotherapy.