Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models

• Published in: Multimedia tools and applications Vol. 82; no. 8; pp. 11365 - 11394
• Main Authors: Gupta, Swadha, Kumar, Parteek, Tekchandani, Raj Kumar
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.03.2023; Springer Nature B.V
• Subjects: 1226: Deep-Patterns Emotion Recognition in the Wild; Accuracy; Classification; Classrooms; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Datasets; Deep learning; Distance learning; Emotion recognition; Emotions; Machine learning; Multimedia Information Systems; Real time; Special Purpose and Application-Based Systems; Students; Teaching methods; Deep learning; Online learner; Facial expressions; Engagement detection; Emotion detection; Real-time engagement detection; Online learning
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-022-13558-9

The dramatic impact of the COVID-19 pandemic has resulted in the closure of physical classrooms and teaching methods being shifted to the online medium.To make the online learning environment more interactive, just like traditional offline classrooms, it is essential to ensure the proper engagement of students during online learning sessions.This paper proposes a deep learning-based approach using facial emotions to detect the real-time engagement of online learners. This is done by analysing the students’ facial expressions to classify their emotions throughout the online learning session. The facial emotion recognition information is used to calculate the engagement index (EI) to predict two engagement states “Engaged” and “Disengaged” . Different deep learning models such as Inception-V3, VGG19 and ResNet-50 are evaluated and compared to get the best predictive classification model for real-time engagement detection. Varied benchmarked datasets such as FER-2013, CK+ and RAF-DB are used to gauge the overall performance and accuracy of the proposed system. Experimental results showed that the proposed system achieves an accuracy of 89.11%, 90.14% and 92.32% for Inception-V3, VGG19 and ResNet-50, respectively, on benchmarked datasets and our own created dataset. ResNet-50 outperforms all others with an accuracy of 92.3% for facial emotions classification in real-time learning scenarios.