Automatic temporal ranking of children’s engagement levels using multi-modal cues

• Published in: Computer speech & language Vol. 50; pp. 16 - 39
• Main Authors: Kim, Jaebok, Truong, Khiet P., Evers, Vanessa
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.07.2018
• Subjects: Children conversation; Engagement; Multi-modal; Non-verbal; Engagement; Play; Children conversation; Multi-modal; Non-verbal
• ISSN: 0885-2308; 1095-8363
• DOI: 10.1016/j.csl.2017.12.005

•We present a novel annotation scheme representing relative levels of children’s engagement with a fine time resolution (each 5 s).•We present temporal dynamics of children’s engagement and relations between engagement and non-verbal features.•We present a robust feature extraction of non-verbal cues, turn-taking and body movement, which deals with a naturalistic group-play setting.•We present a novel ranking algorithm that integrates ranking characteristics and temporal dynamics in a seamless way, which yielded the best performance in our experiments. As children of ages 5–8 often play with each other in small groups, their differences in social development and personality traits usually cause various levels of engagement among others. For example, one child may just observe without engaging at all with others while another child may be interested in both the other children as well as the activity. To develop child-friendly interaction technology such as social robots that can adapt robot behaviours to the social situation of a group of children and facilitate harmonious engagement, we aim to study how we can automatically detect these children’s engagement levels. In this paper, we present a novel automatic method that ranks children in a group according to their engagement level in a temporal way based on non-verbal cues that are robust in naturalistic group settings. Our method combines the omission probability of each rank transformed from discriminative outputs from an SVM ranking method and the transition probability between ranks in time. In comparing our proposed method to other existing methods (such as rule-based ranking, basic SVM, SVM ordinal regression, SVM ranking, and SVMHMM), we found that our novel method yields promising results.