Deep learning network model based on fusion of spatiotemporal features for action recognition

• Published in: Multimedia tools and applications Vol. 81; no. 7; pp. 9875 - 9896
• Main Authors: Yang, Ge, Zou, Wu-xing
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.03.2022; Springer Nature B.V
• Subjects: Accuracy; Activity recognition; Artificial neural networks; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Datasets; Deep learning; Feature recognition; Machine learning; Multimedia Information Systems; Special Purpose and Application-Based Systems; Action recognition; Deep learning; CNN; LSTM
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-022-11937-w

In view of the problem that the current deep learning network does not fully extract and fuse spatio-temporal information in the action recognition task, resulting in low recognition accuracy, this paper proposes a deep learning network model based on fusion of spatio-temporal features (FSTFN). Through two networks composed of CNN (Convolutional Neural Networks) and LSTM (Long Short-Term Memory), the time and space information are extracted and fused; multi-segment input is used to process large-scale video frame information to solve the problem of long-term dependence and improve the prediction accuracy; The attention mechanism improves the weight of visual subjects in the network. The experimental verification on the UCF101 (University of Central Florida 101) data set shows that the prediction accuracy of the proposed FSFTN on the UCF101 data set is 92.7%, 4.7% higher than that of Two-stream, which verifies the effectiveness of the network model.