Action Prediction Based on Partial Video Observation via Context and Temporal Sequential Network With Deformable Convolution

• Published in: IEEE access Vol. 8; pp. 133527 - 133540
• Main Authors: Cheng, Keyang, Lubamba, Eric Kasangu, Liu, Qing
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: action classification; action prediction; Autonomous cars; Background subtraction; Classification; Computational modeling; Computer vision; Context modeling; Convolution; Data models; deformable convolution; Deformable models; Feature extraction; Formability; Predictive models; sequential recurrent network; Subtraction
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2020.3008848

Predicting activity motion form video is of great importance with multiple applications in computer vision. From the self-driving cars field to the health system, the earlier the anticipation the higher the classification probability success. The main challenge of prediction is accurate information of the object of interest in the frame as compared to the full-frame, from the partial observation. To this end, we propose an end-to-end two-stage architecture model that leverages pixel-level features awareness of spatiotemporal information of the object of interest. The first stage of our model is a classification block composed of 3 blocks layers: a background subtraction layer that enables the model to focus on the subject of interest followed by Deformable Convolution layers for feature extraction and finally an additive Softmax for the final classification. Learned information from the first stage is then transferred to the second stage composed of Long Short-Term Memory layers and a final loss function for prediction. The pervasive evaluation on the UT-Interaction, the HMDB51 as well as on the UCF-Sports benchmarks show the betterment of our model performance over threshold probability difference as compared to other solutions. And demonstrate an early action prediction at a lower observation ratio.