Social Scene Understanding: End-to-End Multi-person Action Localization and Collective Activity Recognition

• Published in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3425 - 3434
• Main Authors: Bagautdinov, Timur, Alahi, Alexandre, Fleuret, Francois, Fua, Pascal, Savarese, Silvio
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.07.2017
• Subjects: Activity recognition; Feature extraction; Image sequences; Proposals; Recurrent neural networks
• ISSN: 1063-6919
• DOI: 10.1109/CVPR.2017.365

We present a unified framework for understanding human social behaviors in raw image sequences. Our model jointly detects multiple individuals, infers their social actions, and estimates the collective actions with a single feed-forward pass through a neural network. We propose a single architecture that does not rely on external detection algorithms but rather is trained end-to-end to generate dense proposal maps that are refined via a novel inference scheme. The temporal consistency is handled via a person-level matching Recurrent Neural Network. The complete model takes as input a sequence of frames and outputs detections along with the estimates of individual actions and collective activities. We demonstrate state-of-the-art performance of our algorithm on multiple publicly available benchmarks.