Student behavior recognition based on multitask learning

• Published in: Multimedia tools and applications Vol. 82; no. 12; pp. 19091 - 19108
• Main Authors: Mo, Jianwen, Zhu, Rui, Yuan, Hua, Shou, Zhaoyu, Chen, Lingping
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.05.2023; Springer Nature B.V
• Subjects: Algorithms; Classrooms; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Evaluation; Multimedia Information Systems; Object recognition; Pose estimation; Special Purpose and Application-Based Systems; Student behavior; Target detection; Track 5: Multimedia and Education; Multitask learning; Student behavior recognition; Pose estimation; Object detection
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-022-14100-7

The assessment of students’ classroom behavior is an important part of classroom teaching evaluation. However, teachers cannot timely, objectively and accurately evaluate the listening status of each student in the class. We offer a multitask classroom behavior recognition method that combines human pose estimation and object detection. First, the target detector extracts the individual region from the keyframe as the network’s input. Then, the multitask heatmap network (MTHN) module extracts the intermediate heatmap of multiscale feature association. The attitude estimation and target detection tasks are constructed by mapping relations to obtain the keypoints and object position information. Finally, the keypoints behavior vector and the metric vector are used to model the behavior, and a classroom behavior detection algorithm based on the fully connected network is designed. Additionally, we created a classroom dataset with pose estimation, objects, and behavior labels. Meanwhile, transfer learning is used to solve the problem of insufficient sample size. After several experiments, we show that the detection accuracy of the proposed multitask learning-based student behavior recognition algorithm reaches more than 90%.