Efficient activity recognition using lightweight CNN and DS-GRU network for surveillance applications

• Published in: Applied soft computing Vol. 103; p. 107102
• Main Authors: Ullah, Amin, Muhammad, Khan, Ding, Weiping, Palade, Vasile, Haq, Ijaz Ul, Baik, Sung Wook
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.05.2021
• Subjects: Activity recognition; Artificial intelligence; Deep learning; GRU; IoT; Machine learning; Pattern recognition; Video big data analytics; Deep learning; Video big data analytics; Machine learning; Activity recognition; Pattern recognition; GRU; Artificial intelligence; IoT
• DOI: 10.1016/j.asoc.2021.107102
• ISSN: 1568-4946

Recognizing human activities has become a trend in smart surveillance that contains several challenges, such as performing effective analyses of huge video data streams, while maintaining low computational complexity, and performing this task in real-time. Current activity recognition techniques are using convolutional neural network (CNN) models with computationally complex classifiers, creating hurdles in obtaining quick responses for abnormal activities. To address these challenges in real-time surveillance, this paper proposes a lightweight deep learning-assisted framework for activity recognition. First, we detect a human in the surveillance stream using an effective CNN model, which is trained on two surveillance datasets. The detected individual is tracked throughout the video stream via an ultra-fast object tracker called the ‘minimum output sum of squared error’ (MOSSE). Next, for each tracked individual, pyramidal convolutional features are extracted from two consecutive frames using the efficient LiteFlowNet CNN. Finally, a novel deep skip connection gated recurrent unit (DS-GRU) is trained to learn the temporal changes in the sequence of frames for activity recognition. Experiments are conducted over five benchmark activity recognition datasets, and the results indicate the efficiency of the proposed technique for real-time surveillance applications compared to the state-of-the-art. •Incorporation of two lightweight technologies for activity recognition.•A novel method for sequential features extraction using optical flow CNN model.•A DS-GRU is presented for learning sequential patterns.•An efficient 48 MB trained model for real-time activity recognition.