Multi-Label Activity Recognition using Activity-specific Features and Activity Correlations

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) Vol. 2021; pp. 14620 - 14630
• Main Authors: Zhang, Yanyi, Li, Xinyu, Marsic, Ivan
• Format: Conference Proceeding Journal Article
• Language: English
• Published: United States IEEE 01.06.2021
• Subjects: Activity recognition; Codes; Computer vision; Correlation; Feature extraction; IEEE activities; Visualization
• ISSN: 1063-6919; 1063-6919; 2575-7075
• DOI: 10.1109/CVPR46437.2021.01439

Multi-label activity recognition is designed for recognizing multiple activities that are performed simultaneously or sequentially in each video. Most recent activity recognition networks focus on single-activities, that assume only one activity in each video. These networks extract shared features for all the activities, which are not designed for multi-label activities. We introduce an approach to multi-label activity recognition that extracts independent feature descriptors for each activity and learns activity correlations. This structure can be trained end-to-end and plugged into any existing network structures for video classification. Our method outperformed state-of-the-art approaches on four multi-label activity recognition datasets. To better understand the activity-specific features that the system generated, we visualized these activity-specific features in the Charades dataset. The code will be released later.