A Continuous Learning Framework for Activity Recognition Using Deep Hybrid Feature Models

• Published in: IEEE transactions on multimedia Vol. 17; no. 11; pp. 1909 - 1922
• Main Authors: Hasan, Mahmudul, Roy-Chowdhury, Amit K.
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.11.2015; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Active learning; activity recognition; Adult entertainment; autoencoder; Computational modeling; Data models; Feature extraction; Feature recognition; Human; hybrid feature model; incremental learning; Labeling; Learning; Machine learning; Multimedia; Networks; Recognition; Training; Transaction processing; Videos
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2015.2477242

Most of the research on human activity recognition has focused on learning a static model, considering that all the training instances are labeled and present in advance, while in streaming videos new instances continuously arrive and are not labeled. Moreover, these methods generally use application- specific hand-engineered and static feature models, which are not suitable for continuous learning. Some recent approaches on activity recognition use deep-learning-based hierarchical feature models, but the large size of these networks constrain them from being used in continuous learning scenarios. In this work, we propose a continuous activity learning framework for streaming videos by intricately tying together deep hybrid feature models and active learning. This allows us to automatically select the most suitable features and take the advantage of incoming unlabeled instances to improve the existing model incrementally. Given the segmented activities from streaming videos, we learn features in an unsupervised manner using deep hybrid networks, which have the ability to take the advantage of both the local hand-engineered features and the deep model in an efficient way. Additionally, we use active learning to train the activity classifier using a reduced amount of manually labeled instances. Retraining the models with a huge amount of accumulated examples is computationally expensive and not suitable for continuous learning. Hence, we propose a method to select the best subset of these examples to update the models incrementally. We conduct rigorous experiments on four challenging human activity datasets to demonstrate the effectiveness of our framework.