EnGauge: Engagement Gauge of Meeting Participants Estimated by Facial Expression and Deep Neural Network

• Published in: IEEE access Vol. 11; pp. 52886 - 52898
• Main Authors: Watanabe, Ko, Sathyanarayana, Tanuja, Dengel, Andreas, Ishimaru, Shoya
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: affective computing; Annotations; Artificial neural networks; Behavioral sciences; camera; Cameras; Classification; Data collection; deep neural network; Engagement classification; facial expression; Feature extraction; Machine learning; Measurement methods; Meetings; Observers; Privacy; social interaction; Support vector machines; Wearable technology
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2023.3279428

Measuring the level of engagement among participants in a meeting is crucial for evaluating collective understanding. While previous studies have utilized multiple sensors, such as wearable devices, to gauge engagement levels in offline environments, the shift to remote meetings due to the COVID-19 pandemic presents new challenges. In this study, we propose a method for measuring student engagement during online meetings using only the built-in web cameras on their devices. We collect high, middle, and low engagement level recording data from 24 students. We decided to collect data using the role-acting approach instead of conventional self-reporting or post-experiment annotation. With the feature extraction based approach, we achieved a classification rate of 46.7%. With a deep learning based approach, we achieved a classification rate of 89.5% using MobileNetV2 for leave-one-participant-out cross-validation, which demonstrated higher accuracy than previous studies. From the model, we implement an application EnGauge and conduct a pilot study. The results demonstrate a new approach to data collection, an optimal engagement level recognition model, and application scenarios.